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Plain language summary 


Blended learning is most effective in increasing evidence-based health care 
competencies of health workers 


E-learning is a useful strategy to increase EBHC knowledge and skills, and when combined with 
face-to-face learning, to increase EBHC attitude and behaviour. 


The review in brief 


Evidence-based health care (EBHC) is decision-making for health care, informed by the best 
research evidence. Doctors, nurses and allied health professionals need to have the necessary 
knowledge and skills to apply EBHC. The use of electronic learning (e-learning) for EBHC training 
is increasing. 

E-learning, compared to no learning, improves EBHC knowledge and skills but not attitudes and 
behaviour. There is no difference in outcomes when comparing e-learning to face-to-face learning. 
Combining e-learning with face-to-face learning (blended learning) has a positive impact on EBHC 
knowledge, skills, attitude and behaviour. 


What did the review study? 


Evidence-based health care (EBHC) involves phrasing questions based on a knowledge gap, 
searching for research that can answer the question, critically appraising and interpreting the 
research, applying the results and auditing the process. Electronic learning (e-learning) has 
become an increasingly popular method of teaching EBHC. 

This review assesses the effectiveness of e-learning of EBHC for increasing EBHC competencies in 
healthcare professionals. The primary outcomes are EBHC knowledge, skills, attitude and 
behaviour. 


What is the aim of this review? 

This Campbell systematic review examines the effectiveness of e-learning in improving 
evidence-based health care knowledge and practice. 
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What studies were included? 


Eligible studies were randomised controlled trials (RCTs), cluster RCTs, non-RCTs, controlled 
before-after studies and interrupted time series of any healthcare professional evaluating any 
educational intervention on EBHC, and that was delivered fully (pure e-learning) or in part 
(blended learning) via an electronic platform compared to no learning, face-to-face learning or 
other forms of e-learning of EBHC. 

The review included 24 trials, comprising 20 RCTs and four non-RCTs, with a total of 3,825 
participants. Participants were medical doctors, nurses, physiotherapists, physician assistants, 
athletic trainers and a combination of professionals at all levels of education. The studies included 
a variety of interventions. 


What are the main findings of this review? 


Compared to no learning, pure e-learning improves EBHC knowledge and skills but not attitudes 
and behaviour. Pure e-learning is no better than face-to-face learning in improving any of the 
primary outcomes. 

Blended learning is better than no learning for improving EBHC knowledge, skills, attitude and 
behaviour; and is better than face-to-face learning in improving attitudes and behaviour. 
Compared to pure e-learning, blended learning improves EBHC knowledge. It is not clear which e- 
learning components are most effective in improving outcomes. 

However, the included studies were of moderate to low quality, with a small number of studies 
included in each analysis, and imprecision and inconsistency of results in all comparisons. 

These shortcomings need to be taken into consideration when interpreting the results. 


What do the findings of this review mean? 


E-learning of EBHC, whether pure or blended, compared to no learning, improves EBHC 
knowledge and skills. There is no difference in these outcomes when comparing e-learning to face- 
to-face learning. Blended learning, which typically comprises multiple interventions, appears more 
effective than other types of learning in improving EBHC knowledge, skills, attitude and behaviour. 

Future research should focus on the different components of e-learning and should adequately 
report on all the intervention components, the educational context and implementation strategies. 


How up-to-date is this review? 


The review authors searched for studies published until May 2016. This Campbell Systematic 
Review was published in March 2017. 
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Executive summary 


BACKGROUND 


It is important that all healthcare professionals acquire the knowledge and skills necessary to make 
healthcare decisions which are informed by the current best research evidence. Evidence-based 
health care (EBHC) typically involves phrasing questions based on a knowledge gap, searching for 
research that can answer the question, critically appraising and interpreting the research, applying 
the results and auditing the process. Electronic learning (e-learning) has become an increasingly 
popular method of teaching EBHC but literature on the effectiveness thereof has not been 
synthesized and it is not clear which e-learning strategies are most useful. 


OBJECTIVES 


The primary objective of this review was to assess the effectiveness of e-learning of EBHC on 
increasing EBHC competencies in healthcare professionals. Secondary objectives were to assess the 
effectiveness of specific dimensions of e-learning in increasing EBHC competencies, to assess how 
educational context influences the effectiveness of EBHC e-learning, and to assess how 
implementation approaches influence the effectiveness of EBHC e-learning. 


SEARCH METHODS 


We searched MEDLINE, EMBASE, ERIC, CINAHL, CENTRAL, SCOPUS, Best Evidence Medical 
Education (BEME), Web of Knowledge, Psyclnfo and dissertation databases (ProQuest) for 
relevant studies (24 May 2016). We examined reference lists of included studies and contacted 
experts in the field. We did not apply any language restrictions. 


SELECTION CRITERIA 


We considered randomised controlled trials (RCTs), cluster RCTs, non-randomised controlled 
trials (non-RCTs), controlled before-after studies (CBAs) and interrupted time series (ITS) of any 
healthcare professional at any level of education, evaluating any educational intervention that 
included any or all of the five steps of EBHC and was delivered fully (pure e-learning) or in part 
(blended learning) via an electronic platform compared to no learning of EBHC, face-to-face 
learning of EBHC or other forms of e-learning of EBHC. The primary outcomes were EBHC 
knowledge, EBHC knowledge and skills, EBHC skills, EBHC attitude and EBHC behaviour. 
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DATA COLLECTION AND ANALYSIS 


Two authors independently screened search results and assessed eligibility of potentially eligible 
studies, extracted data and made judgments about risk of bias. Discrepancies were resolved 
through discussion or consultation of a third author. We contacted study authors in case of missing 
data. Due to high levels of heterogeneity between studies, we pooled results using random-effects 
meta-analysis and reported on the standardized mean differences (SMD) and 95% confidence 
intervals for each outcome. 


RESULTS 


We included 24 studies (20 RCTs and four non-RCTs) with a total of 3825 participants in the 
review. Participants included medical doctors, nurses, physiotherapists, physician assistants, 
athletic trainers and a combination of professionals at all levels of education. E-learning 
interventions were heterogeneous with 17 different intervention components. The interventions of 
five studies included only one component while the remaining interventions comprised various 
components in combination and were considered to be multi-faceted. 

Overall we judged studies to be at moderate to high risk of selection bias and high risk of attrition 
bias. Meta-analyses contained a small number of studies and participants. Results were mostly 
imprecise and inconsistent. Our confidence in the following results is therefore low. 

Primary outcomes 

Pure e-learning vs no learning (3 studies) 

Pure e-learning compared to no learning improved EBHC knowledge (SMD 0.71; 95%CI 0.40 to 
1.01; 1 study, 11=175) an d EBHC attitude (SMD 1.05; 95%CI 0.26 to 1.83; 1 study, 11=29). There was 
no difference between groups for EBHC knowledge and skills (SMD 0.47; 95%CI -0.27 to 1.21; 1 
study; n=2g). 

Blended learning vs no learning (5 studies) 

Blended learning compared to no learning improved EBHC knowledge (SMD 0.20; 95%CI 0.13 to 
0.86; 1 study; n=ii9), EBHC knowledge and skills measured at one month post-intervention (SMD 
0.90; 95%CI 0.42 to 1.38; 2 studies; n=24i) and 3+ months post-intervention (SMD 1.11; 95%CI 
0.80 to 1.42; 2 studies; n=i86) and EBHC behaviour measured at 3+ months post-intervention 
(SMD 0.61; 95%CI 0.21 to 1.01; 1 study; n=ioo). There was no difference between groups for EBHC 
knowledge and skills measured immediately post-intervention (SMD 1.40; 95%CI -0.06 to 2.85; 2 
studies, n=24l), EBHC attitude (SMD 0.17; 95%CI -0.09 to 0.43; 2 studies; n=226), EBHC attitude 
measured at 1 month post-intervention (SMD 0.05; 95%CI -0.34 to 0.44; 2 studies; n=24l) and 3+ 
months post-intervention (SMD 0.32; 95%CI -0.02 to 0.67), and EBHC behaviour measured 
directly post-intervention (SMD 0.06; 95%CI -0.28 to 0.40; 2 studies; n=207) and 1 month post- 
intervention (SMD 0.19; 95%CI -0.19 too.56; 1 study; n=i09). 
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Pure e-learning vs face-to-face learning (6 studies) 

We did not find a difference between groups for EBHC knowledge (SMD -0.03; 95%CI 

-0.26 to 0.20; 5 studies, n=632), EBHC skills (SMD -0.15; 95%CI -0.34 to 0.04; 2 studies; n=457) 

or EBHC attitude (SMD 0.11; 95%CI -0.27 to 0.48; 1 study; n=m). 

Blended learning vs face-to-face learning (5 studies) 

We did not find a difference between groups for EBHC knowledge (SMD 0.28; 95%CI 
-0.23 to 0.79; 1 study; n=l46), EBHC knowledge and skills (SMD -0.22; 95%CI -0.49 to 0.05) and 
EBHC skills (SMD -0.21; 95%CI -0.68 to 0.26). Scores for participants in the blended learning 
group were higher for EBHC attitude (SMD 1.07; 95%CI 0.57 to 1.58; 1 study; n=69) and EBHC 
behaviour (SMD 2.34; 95%CI 1.72 to 2.96; 1 study; n=69). 

Blended learning vs pure e-learning (3 studies) 

Blended learning compared to pure e-learning improved EBHC knowledge (SMD 0.69; 95%CI 0.40 
to 0.99; 2 studies, n=i93). For EBHC skills, results favoured pure e-learning for the non-RCT and 
blended learning for the RCT. There was thus significant heterogeneity between studies and the 
pooled effect showed no difference between groups (SMD -0.53; 95%CI -2.31 to 2.25; 2 studies; 
n=2i8). 

Pure e-learning vs pure e-learning (3 studies) 

We found that the interventions improved EBHC skills (SMD 1.30; 95%CI 0.68 to 1.93; 2 studies; 
11=119). Interventions were heterogeneous. One study compared a DVD containing recorded 
PowerPoints and tutorials, as well as access to online learning material to a standard online 
distance learning programme. The other compared an online journal club with an asynchronous 
discussion list to receiving the articles via email and access to journal articles. 

Secondary outcomes 

Secondary outcomes were poorly reported. Attrition rates of learners were high, but did not differ 
between groups. Four studies reported on satisfaction of learning but results were not conclusive 
and both advantages and disadvantages of both methods of learning were identified. 

We were unable to address the secondary objectives of our review, as included studies provided 
insufficient information on educational context and implementation strategies. Meta-analyses 
generally contained a small number of studies, which prevented us from doing subgroup analyses 
on different dimensions of e-learning. 
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AUTHORS’ CONCLUSIONS 


Our findings suggest that e-learning of EBHC, whether pure or blended, compared to no learning, 
improves EBHC knowledge and skills. We did not find a difference in these outcomes when 
comparing e-learning to face-to-face learning, suggesting that both methods of learning can be 
beneficial. It appears that blended learning, which typically comprises multiple intervention 
components, could be more effective than other types of learning in improving EBHC knowledge, 
skills, attitude and behaviour. These findings need to be considered in light of the limited number 
of studies per outcome in each comparison, risk of bias across studies and heterogeneous 
interventions, as well as inconsistent and imprecise results. 

Future research on EBHC e-learning should focus on the effectiveness of various e-learning 
components and should explicitly report on all the intervention components, educational context 
and implementation strategies. 
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l Background 


l.i THE PROBLEM, CONDITION OR ISSUE 


1.1.1 The need for evidence-based health care competencies 

Evidence-based medicine (EBM), introduced in 1991, has its roots in the field of clinical 
epidemiology and was listed as “one of the 15 greatest medical milestones since 1840 ” in the 
British Medical Journal (Montori & Guyatt, 2008). The most commonly used definition of 
evidence-based medicine (EBM) describes it as “the conscientious, explicit and judicious use of the 
current best evidence in making decisions about the care of individual patients” (Sackett, 
Rosenberg, Gray, Haynes, & Richardson, 1996). It thus requires practitioners to bring together 
external evidence that informs about the effects of new tests, treatments and interventions; clinical 
judgement and expertise of the clinician; and the patient’s clinical state, values, preferences, needs 
and predicament. 

These days, EBM is commonly referred to as evidence-based practice (EBP) or evidence-based 
health care (EBHC), as EBM is not limited to medical doctors, but should be adopted by all 
healthcare practitioners. Practicing EBHC typically involves five steps: i) Formulating an 
answerable question from a healthcare problem; ii) finding the best available evidence applicable 
to the question; iii) critically appraising the evidence for validity, clinical relevance and 
applicability; iv) applying the results of the evidence in the healthcare setting; and v) evaluating the 
performance (Dawes et al., 2005). An important aim of EBHC is that beneficial, effective health 
care practices are adopted and that harmful and ineffective ones are abandoned. Consequently, this 
requires healthcare professionals to recognise their deficiencies in knowledge and to adopt a 
philosophy of life-long learning, which is the backbone of practicing EBHC (Greenhalgh, 1997). 

The importance of the knowledge, skills and attitude learnt through the principles of EBHC are 
also highlighted in the Lancet report on the health professional for the 21 st century (Frenk et al., 
2010), which proposes that healthcare professional training should become transformative. One of 
the fundamental shifts of transformative learning aligns almost perfectly with the steps of EBHC; 
the shift from memorization of facts to “critical reasoning that can guide the capacity to search, 
analyse, assess and synthesise information for decision-making” (Frenk et al., 2010). 

In addition, Glasziou and colleagues (2011) have urged educational institutions to teach medical 
students skills which enable them to become life-long learners so that they are able to combine 
external evidence from research with their own expertise and their patients’ values and 
preferences. They emphasize that teaching EBHC skills should form an integral part of the medical 
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curriculum and be re-iterated and practiced throughout undergraduate and postgraduate training 
(Glasziou, Burls, & Gilbert, 2008; Glasziou, Sawicki, Prasad, & Montori, 2011). 


In an evaluation of an online module of EBM, we have proposed a set of EBHC competencies that 
all healthcare professionals should ideally possess once they graduate (Rohwer, Young, & van 
Schalkwyk, 2013). These competencies comprise five key competencies that mirror the five steps of 
EBHC; and enabling competencies which include basic knowledge of epidemiology and 
biostatistics. Figure 1 is a graphic representation of the EBHC competencies. 

Figure 1: EBHC enabling and key competencies 
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A number of systematic reviews have explored the effects of teaching EBHC to healthcare 
professionals, both at an under- and postgraduate stage. From the results, it is evident that 
teaching EBHC to students and health care professionals leads to increased EBHC knowledge and 
skills (Young, Rohwer, Volmink, & Clarke, 2014). The question is therefore no longer whether we 
should be teaching EBHC; but rather how we should be teaching it. Khan and Coomarasamy 
(2006) have proposed a hierarchy of teaching EBHC, where integrated and interactive teaching is 
seen as the most effective way of teaching EBHC (Khan & Coomarasamy, 2006). This, however, 
only refers to traditional lecture-based or face to face teaching of EBHC and does not include the 
increasingly popular method of electronic learning. 


1.2 THE INTERVENTION 


1.2.1 Electronic learning (e-learning) 

Electronic learning or e-learning (as the term is widely used) strategies have been widely adopted 
by educators around the world and it is not surprising that it has also become a buzz word amongst 
medical educators. E-learning, also called web-based learning, online learning, distributed 
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learning, computer-assisted instruction and internet-based learning, can be defined as the delivery 
of training material via information and communication technology (ICT), including the internet, 
CD-ROM, DVD, smartphones and other media, both inside and outside of the classroom (Frehywot 
et al., 2013; Ruggeri, Farrington, & Brayne, 2013; Ruiz, 2006). 

Pure e-learning refers to the use of e-learning materials only, without any face-to-face classroom 
methods. Although it seems to be synonymous to completely online learning, the difference 
between pure e-learning and fully-online learning refers to the delivery platform. While online 
learning relies on a web-based delivery platform and requires internet access, pure e-learning can 
take place without internet access, for example, by using DVDs to deliver the educational content. 
Blended learning combines e-learning components with other traditional face-to-face, lecture- 
based learning in and outside of the classroom (Frehywot et al., 2013). 

Advantages of e-learning include improved accessibility of educational materials at a time 
convenient to the learner; individualised or personalised learning, where the learner decides on the 
amount, pace and place, which allows personal tailoring of the learning experience and meeting of 
individual learner objectives; lower training costs; collaborative learning environments; the ability 
to track learner activity; and provision of consistent and standardized training to a larger number 
of learners all over the world (Clark, 2002; Ruggeri et al., 2013; Ruiz, 2006; Welsh, Wanberg, 
Brown, & Simmering, 2003). 

Disadvantages of e-learning include costs related to the technologies as well as the staff needed, 
although most costs seem to occur at the development stage of e-learning courses; and social 
isolation or the lack of interaction among the participants of the e-learning activity or course. The 
lack of face-to-face networking and peer interaction can be seen as a huge drawback, making e- 
learning less attractive and less useful. Poor instructional design, technical problems and de- 
individualised instruction can also be regarded as disadvantages of e-learning (Cook, 2007; Welsh 
et al., 2003). 

E-learning is a complex intervention with multiple components and dimensions that interact, in a 
linear or non-linear way, in a specific context (Wong, Greenhalgh, & Pawson, 2010). In the 
literature, different components, levels, dimensions and categories of e-learning have been 
described. 

Ruiz et al. (2006) describe the components of e-learning as being i) the development of content ii) 
management of the content and iii) delivery of the content. Regarding development of content, 
digital learning objects, i.e. “any grouping of digital materials structured in a meaningful way 
and tied to an educational objective”, are the fundamental components of lessons, modules and 
curricula; created with the help of instructional design and pedagogical principles. Examples of 
digital learning objects include tutorials, case-based learning scenarios, hypermedia, simulations 
and game-based learning modules (Ruiz, 2006). 

Management of content encompasses the administrative functions (storing, indexing, cataloguing) 
needed to make content available to learners through portals, repositories, digital libraries, 
learning management systems, search engines and e-Portfolios (Ruiz, 2006). 
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Content can be delivered in a synchronous or asynchronous way. Synchronous delivery of content 
refers to real time, instructor-led learning, where all learners receive information simultaneously 
and communicate and interact directly with other learners by logging onto platforms like virtual 
classrooms (e.g. teleconferencing, internet chat forums, instant messaging). Asynchronous delivery 
of content occurs when the transmission and receipt of information occurs at different time points. 
It can include pre-recorded presentations or podcasts, PowerPoint slides, or even more 
sophisticated applications like simulations. While asynchronous communication is done by means 
of email, online bulletin boards, listservs, newsgroups, wikis or weblogs, it enables learners to 
participate in the learning activity at any time of day, from any desired location (Ruiz, 2006; Welsh 
et al., 2003). 

Ruggeri et al. (2013) describe the different dimensions and attributes of e-learning programmes 
under four headings: Synchronicity (asynchronous vs synchronous), location (same place vs 
distributed), independence (individual vs collaborative) and mode (electronic-only vs blended) 
(Ruggeri et al., 2013). 

Cook (2005) describes four levels of instructional design in computer-based learning: Medium, 
which refers to the mode of delivery of instruction (e.g. textbook, face to face, computer-based, 
television); configuration, referring to the “big picture” differences within a given media format 
(e.g. computer: web-based discussion board, web-based or CD-ROM-based tutorial vs face to face: 
small-group discussion, lecture); instructional method, referring to teaching techniques that 
support learning processes (e.g. learning activities, self-assessment questions, clinical cases, 
simulations, group discussions); and presentation, referring to elements of the given medium that 
enhance the intervention, but do not qualify as instructional methods (e.g. hyperlinks, multimedia, 
font simulation fidelity etc.) Cook (2005). 

We developed a system-based logic model of e-learning of EBHC, based on a template from 
Rohwer et al. (2016) depicting the different components, contextual factors and interactions that 
are needed to achieve the desired outcomes (Figure 2) . In an effort to tease out the characteristics 
and dimensions of e-learning interventions in order to be combined meaningfully, we have only 
listed the broad categories that we thought of as being most important. 
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Figure 2: Systems-based logic model of e-learning of EBHC 


Participants 

• Type of healthcare 
worker (e.g. medical 
doctor, Nurse, 
Physiotherapist etc.) 

• Level of education 
(undergraduate, 
postgraduate, CME) 


Educational context 
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Location where learning 

takes place 

• Same place vs. 
distributed 
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university, library, 
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Learner context 

• Background knowledge 
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• Computer literacy 

• Learning style 

• Motivaton 

Institutional context 

• Structure of course 
within larger curriculum 

• Role models 

Socio-economic context 

• Access to internet 

• Access to information 
(databases and 
electronic journals) 

• Affordability 

• Availability of electricity 

• Availability of personal 
computers 


Intervention 

Theory 

Adult learning theory: 

• Self-motivation 

• Personalised learning 

• Distributed learning 


Intervention design 

Components: 

• Course, module, curriculum, workshop on EBHC 

• Learning objectives and content of educational activity 

• EBHC enabling competencies (epidemiology, 
biostatistics, basic searching skills, critical 
thinking) 

• EBHC key competencies (asking questions, 
accessing literature, criticallyappraising 
literature, applying results, evaluating the 
process) 

• Multifaceted intervention vs. Single intervention 

Execution: 

• Duration (6 weeks, one year etc) 

• Intensity (e.g. 2 hours) 

• Dose (e.g. twice a week; once a month) 

• Timing (within study programme etc.) 

• Integrated or stand-alone 


Intervention delivery 
Dimensions: 

• Pure e-learning vs. Blended learning 

• Collaborative (interactive) vs. Individual learning 

• Synchronous vs. Asynchronous delivery 

Delivery agent: 

• Facilitators and tutors: Attitude, communication skills, 
teaching skills, engagement with learners 

Organisation and structure: 

• Institutions offering educational activity (cost, capacity, 
culture) 


Outcomes 

Intermediate outcomes 
Process outcomes 

Barriers to method of teaching EBHC 
Enablers of method of teaching EBHC 
Learner satisfaction 
Teacher satisfaction 
Cost 
Attrition 

Surrogate outcomes 

•EBHC knowledge* 

•EBHC skills* 

•EBHC attitude* 

Behaviour outcomes 

• EBHC behaviour* (e.g. Question 
formulation, reading habits etc) 

• Evidence-based practice 

• Learner adherence 

Non-health outcomes 

• Evidence-based guideline implementation 

• Health care delivery (health systems) 

Health outcomes 

• Individual health outcomes 

• Population health outcomes 


3E 


Healthcare context 


Socio-cultural 

Socio-economic 

Epidemiological 

Legal 

Ethical 

Political 


*Bold outcomes represent primary outcomes, 
the rest refer to secondary outcomes 


1.3 HOW THE INTERVENTION MIGHT WORK 


1.3.1 How e-learning may work 

A number of systematic reviews have been conducted in various fields of medical and health 
science education, examining the effect of different types of e-learning on knowledge, skills and 
behaviour. 

A systematic review conducted by the WHO evaluated the effectiveness of e-learning for 
undergraduate health professions education (Al-Shorbaji, Atun, Car, Majeed, & Wheeler, 2015). 
They included 209 studies and concluded that e-learning is similar to traditional face-to-face 
learning in terms of knowledge and skill acquisition. The majority of included studies were judged 
to be at high risk of bias. 

Cook et al. (2008) conducted a systematic review to determine the effectiveness of internet-based 
learning for health professionals compared to no intervention or non-internet interventions. The 
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authors pooled data in a random-effects meta-analysis and found significant results favoring 
internet-based learning for knowledge, skills, as well as behaviour and patient effects when 
compared to no learning. But when internet-based learning was compared to alternative methods 
of learning, there was no difference in results between groups. In a subsequent analysis on 
instructional design variations in internet-based learning, the authors concluded that interactivity, 
practice exercises, repetition and feedback appeared to be associated with improved learning 
outcomes (Cook et al., 2010). A review by Booth and colleagues (2009) found that presentation and 
design, flexibility, peer communication, support and knowledge validation were effective e-learning 
techniques for enhancing the learning experience of students. 

Liu et al. (2016) conducted a systematic review on the effectiveness of blended learning compared 
to no or other learning amongst health professionals. They included 56 studies and results favored 
blended learning when compared to no learning and to other learning. However, authors caution 
that high levels of heterogeneity should be taken into account when interpreting results. 

Another recent systematic review assessed the effectiveness of internet-based e-learning on health 
care professional behaviour and patient outcomes (Sinclair, Kable, Levett-Jones, & Booth, 2016). 
The authors found 7 studies that assessed behaviour of healthcare professionals and no studies that 
reported on patient outcomes. They were unable to pool results in meta-analysis due to substantial 
heterogeneity between studies. Although results from individual studies showed that e-learning 
was just as effective as face-to-face learning and better than no learning, authors concluded that 
there was currently insufficient evidence to answer their question. 

Other studies have examined how e-learning works, focusing more on the essential components for 
successful learning, as well as certain contextual factors that influence learning. A realist review by 
Wong et al. (2010), looking at “what works, for whom and in what circumstances” when 
considering internet-based medical education, concluded that online courses need to engage the 
learners to use the technology, which is more likely to happen when the technology is perceived as 
being useful and easy to use; and that interactivity should be a key characteristic of an online 
course, since learners want to be able to enter a dialogue with tutors and peers (Wong et al., 2010). 

E-learning technologies afford a new learner paradigm based on the adult learning theory, 
explaining that adults learn by relating new information to past experiences; tailoring learning to 
their unique needs (open learning); and applying learning in practice, resulting in more effective 
and efficient learning experiences. The attributes of accessibility and convenience (distributed 
learning) personalise the learning, because students decide when and where they are receptive to 
learning. Learning becomes an individual experience where adults learn because they want to learn 
- not because they are told to learn. This shift from “expert-led teaching to user-lead learning” 
results from intrinsic motivation and offers a much stronger learning stimulus. Some evidence 
suggests that learning by means of e-learning is more efficient and that learners gain knowledge 
faster, which in turn translates back to improved motivation and enhanced learning. (Clark, 2002; 
Dabbagh, 2005; Ruggeri et al., 2013; Ruiz, 2006). 

When considering these attributes and pedagogical underpinnings of e-learning, one can argue 
that these fit perfectly within the EBHC paradigm. Enhanced learning occurs through internal 
motivation, rather than external drivers and requires acknowledgement of shortcomings 
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(knowledge gaps) and adoption of a reflective approach towards one’s own practice. This alignment 
between attributes of e-learning and EBHC foci suggests that making use of e-learning in this 
context could be of value. 

The pathway from EBHC e-learning activities to the desired outcomes, i.e. increased knowledge, 
skill, attitude and behaviour, as well as health outcomes is depicted in Figure 3, a process-oriented 
logic model, based on the template by Rohwer et al. (2016). 

Figure 3: Process-oriented logic model showing the pathway from EBHC learning to 
desired outcomes 



It is important to note that this pathway does not take place in a vacuum, but that ultimately, 
improved health care delivery and improved health outcomes do not only rely on evidence-based 
practice, but are influenced by a variety of other factors within the healthcare context e.g. socio- 
cultural, socio-economic, epidemiological, legal, ethical and political factors on a national and 
international level (See Figure 2). 


1.4 WHY IT IS IMPORTANT TO DO THE REVIEW 


A number of systematic reviews assessing the effectiveness of e-learning for healthcare 
professionals at various levels have been published in recent years. As described above, these 
reviews assessed e-learning of any content related to health professions education (Al-Shorbaji et 
al., 2015; Cook et al., 2008; Liu et al., 2016; Sinclair et al., 2016). Generally, systematic reviews 
concluded that e-learning was better than no learning, but neither superior nor inferior to other 
methods of learning. Although these reviews do not represent a comprehensive list of all published 
reviews in this area, they are an indication of the current interest in this field. 
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When examining the literature on EBHC learning, we also found a number of recently published 
studies. A recent overview of systematic reviews (Young et al., 2014) that included 16 systematic 
reviews examining the effects of educational activities on EBHC, found that when comparing single 
interventions (a workshop, journal club, lecture or e-learning) with multifaceted interventions (a 
combination of different strategies e.g. lectures, tutorials, e-learning, journal clubs, etc.) 
multifaceted clinically integrated educational activities were more likely to increase EBHC 
knowledge, skills, attitude and behaviour. Although some of the studies included in the systematic 
reviews related to e-learning activities, it is still unclear whether e-learning of EBHC leads to 
increased knowledge, skills, attitude and behaviour. The need to synthesise studies specifically 
evaluating the effect of EBHC e-learning was therefore identified. 

Another recent systematic review (Ilic & Maloney, 2014) that aimed to determine what type of 
educational methods was most effective in increasing EBHC knowledge and skills found that there 
was no difference between the various educational methods for EBHC. It is therefore still unknown 
whether some e-learning strategies are more useful and effective than others. 

This review considers e-learning of EBHC to be a complex intervention and aims to address the 
effectiveness of the interventions, as well as identify the implementation and contextual factors 
that are important in delivering the intervention. In teasing out these issues, we hope to be able to 
formulate evidence-based recommendations for EBHC teachers and program developers. 
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2 Objectives 


Primary objective: 

• To assess the effectiveness of e-learning of EBHC on increasing EBHC competencies in 
healthcare professionals 

EBHC competencies include (Figure if 

Enabling competencies: biostatistics, epidemiology, basic searching skills, philosophy of 
critical enquiry 

Key competencies: asking clear questions, accessing the literature to find the best available 
evidence, critically appraising the evidence for validity and interpreting results, applying 
the results, and auditing the process 

Secondary objectives: 

• To assess the effectiveness of specific dimensions of e-learning in increasing EBHC 
competencies 

• To assess how educational context influences the effectiveness of EBHC e-learning 

• To assess how implementation approaches influence the effectiveness of EBHC e-learning 


19 


The Campbell Collaboration | www.campbellcollaboration.org 


3 Methods 


The protocol for this systematic review has been published in the Campbell Library (Rohwer, 
Rehfuess, & Young, 2014). 


3.1 CRITERIA FOR CONSIDERING STUDIES FOR THIS REVIEW 


3.1.1 Types of studies 

We followed the guidance of the Effective Practice and Organisation of Care (EPOC) review group 
of Cochrane. We included randomised controlled trials (RCTs), cluster randomised controlled 
trials and non-randomised controlled trials (non-RCTs) in the review. In addition, we considered 
controlled before-after studies (CBAs) with at least two intervention and two control sites; and 
interrupted time series (ITS) with a clearly defined point in time when the intervention occurred 
with at least three data points before and three data points after the intervention. 

In addition, we included studies supporting the included intervention studies that yielded 
information on the process of implementation. These studies were either quantitative (e.g. process 
evaluations, quantitative interview studies) or qualitative in nature (e.g. focus groups, qualitative 
interview studies). The information gathered was either reported within the above included 
intervention studies, or cited within these and published separately. We did not include these 
supporting studies in the data synthesis, but rather reported relevant results separately. 

We excluded studies without a comparison group. 

3.1.2 Types of participants 

We included all healthcare professionals, including doctors, dentists, nurses, occupational 
therapists, physiotherapists, dieticians, audiologists, mental health professionals, psychologists, 
counsellors, social workers at undergraduate, postgraduate (e.g. residents) or continuing medical 
education (CME) level; working in primary, secondary or tertiary environments. 

3.1.3 Types of interventions 
3. 1.3.1 Interventions 

We included any educational intervention (a co-ordinated educational activity) that included any 
or all of the 5 steps of EBHC (asking questions, searching the literature, critically appraising the 
literature, applying the results, evaluating the process) and was delivered via an electronic platform 
(e-learning only), or made use of e-learning in a supplementary way (blended learning). E-learning 
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(web-based learning, online learning, distributed learning, computer-assisted instruction) was 
regarded as: delivery of training material via any electronic media (internet, CD-ROM, DVD, 
smartphones and other media), independent of the delivery platform. 

3. 1.3. 2 Comparisons 

Comparisons included no educational intervention (no EBHC learning); any educational 
intervention that included any or all of the 5 steps of EBHC (asking questions, searching the 
literature, critically appraising the literature, applying the results, evaluating the process) that was 
delivered via an electronic platform, but with different components than the intervention (e.g. 
interactivity vs no interactivity; synchronicity vs asynchronicity); and any educational intervention 
that included any or all of the 5 steps of EBHC (asking questions, searching the literature, critically 
appraising the literature, applying the results, evaluating the process) and was delivered via face- 
to-face learning, with no e-learning components. 

3.1.4 Types of outcome measures 

Primary outcomes 

• EBHC knowledge (as measured by study authors, for example, evaluating knowledge scores 
with a validated pre-and post-training questionnaire such as Fresno test (Ramos, Schafer, & 
Tracz, 2003) or Berlin test (Fritsche, Greenhalgh, Falck-Ytter, Neumayer, & Kunz, 2002) ) 

• EBHC knowledge and skills measured as a composite outcome as measured by study authors 

• EBHC skills (as measured by study authors, for example, evaluating skill scores with a validated 
pre-and post-training questionnaire such as Fresno test (Ramos et al., 2003) or Berlin test 
(Fritsche et al., 2002)) 

• EBHC attitude (as measured by study authors, for example, with Likert-scale questions pre- 
and post-training) 

• EBHC behaviour (as measured by study authors, for example, self-reported behaviour changes, 
amount of questions formulated, amount of searches done; or more objective measurements, 
for example, proportion of clinical cases where evidence was consulted) 

Secondary outcomes 

• Process outcomes 

o Satisfaction of students with method of learning (as measured by study authors, for 
example, Likert-scale questions post-training, or open-ended questions yielding 
qualitative data) 

o Satisfaction of educators with method of learning (as measured by study authors, 
for example, with Likert-scale questions post-training, or open-ended questions 
yielding qualitative data) 
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o Enablers of the method of learning EBHC (as measured by study authors, for 
example, with Likert-scale questions post-training, or open-ended questions 
yielding qualitative data) 

o Barriers to the method of learning EBHC (as measured by study authors, for 
example, with Likert-scale questions post-training, or open-ended questions 
yielding qualitative data) 

o Cost (a sum of all the monetary cost involved in the training) 

o Attrition of learners (measured quantitatively by looking at the proportion of 
learners that complete the training) 

• Behaviour outcomes 

o Learner adherence (measured either quantitatively by looking at the proportion of 
learners that adhere to the prescribed learning, or qualitatively by asking learners 
whether about their learning experience) 

o Evidence-based practice (measured qualitatively through self-report of 
practitioners) 

• Non-health outcomes 

o Evidence-based guideline implementation (for example, measured quantitatively 
by looking at practice audits) 

o Health care delivery (measured qualitatively through self-report of practitioners) 

• Health outcomes 

o Individual health outcomes (these refer to clinical patient-orientated outcomes, 
measured quantitatively) 

o Population health outcomes (these refer to health outcomes of the community or 
the entire population measured quantitatively, for example, disease incidence rates) 


3.2 SEARCH METHODS FOR IDENTIFICATION OF STUDIES 


3.2.1 Electronic searches 

We developed a comprehensive search strategy consisting of relevant terms (Table Q.l) and 
searched electronic databases (MEDLINE via PubMed, EMBASE via Ovid, ERIC, CINAHL, 
CENTRAL, SCOPUS, Best Evidence Medical Education (BEME), Web of Knowledge, Psyclnfo) and 
dissertation databases (ProQuest) up to 24 May 2016 for relevant studies. We did not apply any 
language restrictions. 
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3.2.2 Searching other resources 


We complemented our search with a thorough examination of reference lists of identified studies 
and contacted experts in the field to identify any on-going or unpublished studies. We also 
searched trial registries (ICTRP) for on-going studies (24 May 2016). 


3.3 DATA COLLECTION AND ANALYSIS 


3.3.1 Selection of studies 

Two authors (AR and either TY, ER or NVM) independently screened the search outputs and 
abstracts for relevant studies. We retrieved full texts of studies with seemingly relevant abstracts 
and two authors (AR and TY, ER or NVM) independently assessed them for eligibility using the 
pre-specified inclusion criteria. We resolved discrepancies through discussions and consultations 
with a third author before classifying studies as included, excluded, or on-going. 

3.3.2 Data extraction and management 

Two authors (AR and either NVM, TY or ER) extracted data from relevant studies independently 
and in duplicate using piloted, electronic data extraction forms. Discrepancies were resolved 
through discussions and consultations with a third author. We contacted authors in case of missing 
data. 

We extracted data related to study design (type of study, duration of study, country), participants 
(number of participants, type of health care professionals, level of education), interventions 
(learning theory, educational content, duration, intensity, dose and timing of intervention, delivery 
of interventions), outcomes (primary and secondary outcomes, measurement details and time 
point at which outcomes were measured), results (measures of effect with 95% confidence 
intervals, qualitative data) and educational context (setting, learner context, institutional context 
and socio-economic context). We considered the term “learning theory” in a broad way, to describe 
a body of implicit or explicit ideas about how learning works (May et al., 2007; Wells, Williams, 
Treweek, Coyle, & Taylor, 2012). We extracted theories, models or learning approaches, as defined 
by study authors. 

We broadly categorised interventions and controls into pure e-learning, blended learning, face-to- 
face learning or no learning. In addition, we indicated whether interventions were single or multi- 
faceted interventions, whether EBHC learning was stand-alone or integrated, and for all e-learning 
interventions, whether they required individual or collaborative learning. 

In addition, we compiled a matrix of the intervention components related to the included 
interventions and comparisons in order to compare the components across studies. The matrix was 
informed by components identified from the included studies and grouped according to mode of 
delivery, for example, e-learning components and face-to-face components. 
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3.3*3 Assessment of risk of bias in included studies 

For each included study, two authors (AR and either NVM, TY, or ER) independently assessed the 
risk of bias according to the Cochrane Effective Practice and Organisation of Care’s (EPOC) 
suggested risk of bias criteria (EPOC, 2015) for the following domains: selection bias, performance 
bias, detection bias, attrition bias, contamination (when participants in the comparison group are 
exposed to the intervention and vice versa), reporting bias, other bias (Table Q.2) . 

In addition, we assessed the possibility of additional biases for cluster RCTs according to the 
Cochrane handbook (Higgins & Green, 2008). 

• Recruitment bias: We described whether participants were recruited before or after 
randomization of clusters. We regarded studies as having low risk of recruitment bias if 
participants were recruited before randomization of clusters; high risk of bias if they were 
recruited after randomization; and unclear risk of bias if information about the timing of 
recruitment is unclear. 

• Baseline imbalance: We described any baseline imbalances between individuals and 
clusters 

• Loss of clusters: We described the number of clusters lost as well as reasons for attrition 

• Incorrect analysis: We described whether analysis was adjusted for clustering 

• Compatibility with RCTs randomised by individuals: We described whether the 
intervention effects may be systematically different from individually randomised 
controlled trials i.e. whether it was likely that the effect size was over- or underestimated. 

We resolved discrepancies through discussion and consultation with a third author if needed. 

3*3*4 Measures of treatment effect 

3. 3. 4.1 Dichotomous data 

We presented rates of attrition of learners as risk ratios (RR) with their respective 95% confidence 
intervals (Cl). 

3.3.4.2 Continuous data 

Continuous outcomes included EBHC knowledge and skill scores; EBHC attitude scores and EBHC 
behaviour. We presented continuous data as the standardised mean difference (SMD) with 95% 
confidence intervals, to take into account differences related to various measurement tools. We 
interpreted the size of the effect as follows, based on Sawilowsky (2009): 

• SMD 0.01-0.19: very small effect 

• SMD 0.2-0.49: small effect 

• SMD 0.5-0.79: medium effect 

• SMD 0.8-1.19: large effect 
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• SMD 1.2-1.99: very large effect 

• SMD 2.0 and above: huge effect 

3'3'4'3 Other data 

Outcomes not measured numerically were reported narratively. These outcomes included EBHC 
behaviour, satisfaction with the method of learning, and barriers and enablers to the method of 
learning EBHC. 

3 * 3*5 Unit of analysis issues 

In cluster RCTs, where authors had appropriately adjusted results for clustering, we included the 
adjusted effect estimates and standard errors using the generic inverse-variance method in 
RevMan. For outcomes where authors had not appropriately adjusted for clustering, we adjusted 
the data by calculating the ‘effective sample size’ of each intervention group in the cluster RCT. 
This was done by dividing the original sample size of an intervention group by the design effect 
(Higgins & Green, 2008). We then included the adjusted data in the meta-analysis. 

For multi-arm studies, which contributed multiple comparisons to a particular meta-analysis, we 
split the ‘shared’ group (e.g. two intervention groups and one control group) as appropriate to 
avoid the inclusion of data from the same patient more than once in the same analysis. 

3.3.6 Dealing with missing data 

We contacted authors in case of missing data. Where authors did not respond, we calculated the 
means and standard deviations (SD) according to the methods described in the Cochrane 
Handbook (Higgins & Green, 2008). The mean difference (MD) and SD were then converted to a 
SMD, standard error (SE) and corresponding 95% confidence intervals (CIs) with RevMan 
software (RevMan, 2011). 

For continuous outcomes, we only included participants for whom outcomes were measured, i.e., 
per-protocol analysis. For rates of attrition of learners, we used intention to treat analysis, i.e., we 
included all participants that were randomised to a specific group as the denominator in the 
analysis. 


3.3.7 Assessment of heterogeneity 

We assessed heterogeneity by describing variability among studies regarding participants, 
interventions and outcomes (educational heterogeneity), as well as variability in study design and 
risk of bias (methodological heterogeneity). 

Educational heterogeneity was explored by clearly documenting the characteristics of participants; 
all components of the intervention relating to intervention design and delivery; as well as outcomes 
and measurement of outcomes in table format. In addition, the educational context (setting, 
learner context, institutional context, socio-economic context), in which the intervention was 
delivered, was explored and reported. See Figure 2. 
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Methodological heterogeneity was explored by clearly documenting study design as well as risk of 
bias for each study. 

We assessed statistical heterogeneity by looking at the I 2 statistic as well as the Chi 2 test statistic. 
We considered an I 2 value of more than 30%, and a p-value of less than 0.10 for the Chi 2 test for 
heterogeneity to be an indicator of important heterogeneity. In case of considerable heterogeneity, 
we performed random-effects meta-analysis instead of fixed-effect meta-analysis or reported 
outcomes in tabular or narrative format. 

3.3,8 Assessment of reporting biases 

We did not assess reporting biases with funnel plots, since we had less than 10 included studies per 
outcome. 


3.3.9 Data synthesis 

Due to important heterogeneity related to the participants, interventions and outcome 
measurements, we used random-effects meta-analysis to pool results using RevMan. For 
continuous outcomes, we converted means and standard deviations reported at follow-up to SMD 
and SEs, and reported the pooled SMD with 95% CIs. For studies where outcomes were reported as 
mean change scores, we calculated follow-up means from baseline means and used baseline SDs 
for follow-up SDs before converting to SMDs (Higgins & Green, 2008). For dichotomous 
outcomes, we reported the pooled risk ratio (RR) with 95% CIs. Studies were grouped under six 
main comparisons: 

1. Pure e-learning vs no learning 

2. Blended learning vs no learning 

3. Pure e-learning vs face-to-face learning 

4. Blended learning vs face-to-face learning 

5. Blended learning vs pure e-learning 

6. Pure e-learning vs pure e-learning 

3.3.10 Subgroup analysis and investigation of heterogeneity 
We performed the following subgroup analysis on primary outcomes: 

1. Study design: Non-RCTs vs RCTs 

3.3.11 Sensitivity analysis 

We did not perform sensitivity analysis on primary outcomes, as the number of included studies 
per outcome was very small. 
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4 Results 


4.1 DESCRIPTION OF STUDIES 


4,1.1 Results of the search 

Our search of the databases yielded 12980 citations (24 May 2016) and we identified three further 
citations from screening references of included studies and experts. After removal of duplicates, we 
screened 6175 titles and abstracts, of which we identified 55 as being potentially eligible. We 
assessed eligibility of these by obtaining and screening the full texts of the studies. We included 24 
studies f Table Q.2I . classified one study as ongoing (Table Q.4I and excluded 30 studies (Table Q.fO 
(Figure 4). 

Figure 4: Flow diagram of included studies 
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4 .1.2 Included studies 


We included 24 studies with a total of 3825 participants in the review. Details of individual studies 
can be found in the table of characteristics of included studies (Table 9.3) . Table q.6 summarises 
the characteristics of included studies. 

4.1.2.1 Study designs 

Thirteen included studies were individually randomised controlled trials (Bergold et al., 2013; 
Bradley et al., 2005; Brettle & Raynor, 2013; Brouwers et al., 2011; Davis et al., 2007; Davis et al., 
2008; Dizon et al., 2014; Forsetlund et al., 2003; Horiuchi et al., 2009; Kamin et al., 2001; Macrae 
et al., 2004; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014), seven studies were 
cluster RCTs (Hadley et al., 2010; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 
2012; Laibhen-Parkes, 2014; McLeod et al., 2010) and four were quasi-randomised trials 
(Fernandez et al., 2014; Ilic et al., 2013; Ramos-Morcillo et al., 2015; Schilling et al., 2006). We did 
not identify any eligible CBAs or ITS. 

4.1.2.2 Participants 

Participants of 14 included trials were medical doctors (Bergold et al., 2013; Bradley et al., 2005; 
Davis et al., 2007; Davis et al., 2008; Forsetlund et al., 2003; Hadley et al., 2010; Ilic et al., 2013; 
Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 2012; Macrae et al., 2004; 

McLeod et al., 2010; Schilling et al., 2006). Of these, five trials included undergraduate students 
(Bradley et al., 2005; Davis et al., 2008; Ilic et al., 2013; Ilic et al., 2015; Schilling et al., 2006), six 
trials included qualified clinicians (Bergold et al., 2013; Davis et al., 2007; Forsetlund et al., 2003; 
Hadley et al., 2010; Kok et al., 2013; Macrae et al., 2004), and three trials included residents; that 
is, postgraduate students with a basic medical degree who are specialising (Kulier et al., 2009; 
Kulier et al., 2012; McLeod et al., 2010). Six trials included nurses (Brettle & Raynor, 2013; 
Fernandez et al., 2014; Horiuchi et al., 2009; Laibhen-Parkes, 2014; Ramos-Morcillo et al., 2015; 
Saunders et al., 2016), of which one study included undergraduate (Brettle & Raynor, 2013) and 
one postgraduate (Fernandez et al., 2014) students; and four trials included practicing nurses. The 
remaining four trials included practicing physiotherapists (Dizon et al., 2014), undergraduate 
physician assistants (Kamin et al., 2001), athletic trainers (Welch, Van Lunen, & Hankemeier, 

2014) and a combination of healthcare professionals, clinicians, methodologists, policy makers and 
trainees (Brouwers et al., 2011). 

4. 1.2. 3 Location of studies 

Most studies were conducted in high-income countries, four in the UK (Brettle & Raynor, 2013; 
Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010), four in the USA (Kamin et al., 2001; 
Laibhen-Parkes, 2014; McLeod et al., 2010; Schilling et al., 2006) two in Norway (Bradley et al., 
2005; Forsetlund et al., 2003), two in Canada (Brouwers et al., 2011; Macrae et al., 2004), one each 
in Germany (Bergold et al., 2013), The Netherlands (Kok et al., 2013), Spain (Ramos-Morcillo et 
al., 2015), Finland (Saunders et al., 2016), Australia (Ilic et al., 2013) and Japan (Horiuchi et al., 
2009). Three studies were conducted in more than one high-income country, one in the UK and 
The Netherlands (Kulier et al., 2009), one in Australia and Hong Kong (Fernandez et al., 2014), 
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and one in Australia and Malaysia (Ilic et al., 2015). The remaining two studies were conducted in 
low-and middle-income countries, one in the Philippines (Dizon et al., 2014) and one had multiple 
study sites in Argentina, Brazil, Democratic Republic of Congo, India, Philippines, South Africa and 
Thailand (Kulier et al., 2012). One study did not report where participants were based, since they 
were members of the national athletics trainers’ association (NATA) which is a worldwide 
association (Welch, Van Lunen, & Hankemeier, 2014). Attempts to contact the authors for 
additional information were not successful. 

4.1.2.4 Interventions 

All but two studies had two study arms. One study had three arms (Brouwers et al., 2011) and one 
had four study arms (Fernandez et al., 2014). Interventions were heterogeneous and are described 
in detail in the table of characteristics of included studies (Table q.A) . with the intervention 
components summarised in Table Q. 7 . 

Thirteen studies (Bradley et al., 2005; Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; 
Fernandez et al., 2014; Hadley et al., 2010; Horiuchi et al., 2009; Kamin et al., 2001; Laibhen- 
Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; Schilling et al., 2006; Welch, Van Lunen, & 
Hankemeier, 2014) included interventions that were pure e-learning interventions, while eleven 
studies evaluated blended learning (Bergold et al., 2013; Brettle & Raynor, 2013; Dizon et al., 2014; 
Fernandez et al., 2014; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Kulier 
et al., 2012; Ramos-Morcillo et al., 2015; Saunders et al., 2016). Amongst the pure e-learning 
interventions, ten required individual learning only (Bradley et al., 2005; Brouwers et al., 2011; 
Davis et al., 2007; Davis et al., 2008; Fernandez et al., 2014; Hadley et al., 2010; Horiuchi et al., 
2009; Laibhen-Parkes, 2014; Welch, Van Lunen, & Hankemeier, 2014) and four incorporated 
collaborative learning in the intervention (Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 
2010; Schilling et al., 2006). Amongst the blended learning interventions, six required individual 
learning (Bergold et al., 2013; Brettle & Raynor, 2013; Dizon et al., 2014; Kulier et al., 2009; Kulier 
et al., 2012; Saunders et al., 2016) and five included collaborative learning activities (Forsetlund et 
al., 2003; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 2013; Ramos-Morcillo et al., 2015). Four of the 
pure e-learning interventions delivered the content via CD-ROM or DVD (Bradley et al., 2005; 
Davis et al., 2007; Davis et al., 2008; Fernandez et al., 2014), whereas all other interventions were 
delivered asynchronously via an online learning platform or email. 

In eight studies the interventions were regarded as single-component interventions and delivered 
as stand-alone teaching (Bradley et al., 2005; Brettle & Raynor, 2013; Brouwers et al., 2011; Davis 
et al., 2007; Davis et al., 2008; Hadley et al., 2010; Horiuchi et al., 2009; Laibhen-Parkes, 2014; 
Welch, Van Lunen, & Hankemeier, 2014). The remaining interventions were multi-faceted, 
comprising more than one component. Of these, six were delivered as stand-alone teaching 
(Brouwers et al., 2011; Fernandez et al., 2014; Kamin et al., 2001; Macrae et al., 2004; McLeod et 
al., 2010; Ramos-Morcillo et al., 2015), while ten were integrated into clinical practice (Bergold et 
al., 2013; Dizon et al., 2014; Forsetlund et al., 2003; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 
2013; Kulier et al., 2009; Kulier et al., 2012; Saunders et al., 2016; Schilling et al., 2006). 

Intervention components of included studies differed across studies (Table 9.7). The most common 
e-learning component was recorded PowerPoint presentations, which was included in twelve 
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studies (Bergold et al., 2013; Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Fernandez 
et al., 2014; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2015; Kulier et al., 2009; Kulier et 
al., 2012; Laibhen-Parkes, 2014; Schilling et al., 2006), five of which had this as the only 
intervention component (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Hadley et al., 
2010; Laibhen-Parkes, 2014). Eight studies included access to an online teaching site and learning 
materials (Brouwers et al., 2011; Dizon et al., 2014; Fernandez et al., 2014; Forsetlund et al., 2003; 
Ilic et al., 2013; Ilic et al., 2015; Kamin et al., 2001; Ramos-Morcillo et al., 2015), but they all had at 
least one other component. These other e-learning components included online tutorials (Brettle & 
Raynor, 2013; Brouwers et al., 2011; Fernandez et al., 2014; Saunders et al., 2016; Schilling et al., 
2006), online exercises, assignments and clinical scenarios (Bergold et al., 2013; Horiuchi et al., 
2009; Kamin et al., 2001; Kulier et al., 2009; McLeod et al., 2010; Ramos-Morcillo et al., 2015; 
Saunders et al., 2016), online support by a tutor and feedback (Brouwers et al., 2011; Dizon et al., 
2014; Forsetlund et al., 2003; Ramos-Morcillo et al., 2015; Saunders et al., 2016), use of online 
tools such as checklists and calculators (Dizon et al., 2014; Schilling et al., 2006), asynchronous 
discussion lists (Forsetlund et al., 2003; Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 
2010; Ramos-Morcillo et al., 2015), and online journal clubs (Macrae et al., 2004; McLeod et al., 
2010). Receiving an electronic newsletter (Forsetlund et al., 2003), access to databases (Forsetlund 
et al., 2003) and mobile learning at the bedside (Ilic et al., 2015) are intervention components that 
were each included in only one study. Two studies did not clearly describe the intervention 
components only referring to it as an “interactive online course” in Kok et al. (2013), and a “web- 
based module” in Welch, Van Lunen, and Hankemeier (2014). 

Face-to-face intervention components that formed part of blended learning included classroom- 
based as well as clinical activities. The most commonly included classroom-based component was 
an interactive workshop, which was part of the intervention in five studies (Dizon et al., 2014; 
Forsetlund et al., 2003; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 2013). Five studies included 
didactic lectures (Dizon et al., 2014; Forsetlund et al., 2003; Ilic et al., 2015; Ramos-Morcillo et al., 
2015; Saunders et al., 2016), three included small group discussions (Ilic et al., 2013; Ilic et al., 
2015; Kok et al., 2013) and two included hands-on computer-based training (Brettle & Raynor, 
2013; Kok et al., 2013). Clinical activities and assignments formed part of the intervention in four 
studies (Bergold et al., 2013; Ilic et al., 2013; Kulier et al., 2009; Kulier et al., 2012), with three of 
these also including access to a tutor or facilitator in the clinical field (Bergold et al., 2013; Kulier et 
al., 2009; Kulier et al., 2012). 

Content covered by interventions mostly related to EBHC key competencies (Table 9.8). Four 
studies focussed on asking questions and accessing the literature (Brettle & Raynor, 2013; Kamin 
et al., 2001; Ramos-Morcillo et al., 2015; Schilling et al., 2006), while the content of three others 
only related to critical appraisal (Brouwers et al., 2011; Macrae et al., 2004; McLeod et al., 2010). 
The learning content of one study related to the first three key competencies (asking questions, 
accessing the literature and critical appraisal) (Hadley et al., 2010), while the remaining studies 
included the four key competencies: asking questions, accessing the literature, critical appraisal 
and applying the results (Bergold et al., 2013; Bradley et al., 2005; Davis et al., 2007; Davis et al., 
2008; Dizon et al., 2014; Fernandez et al., 2014; Forsetlund et al., 2003; Horiuchi et al., 2009; Ilic 
et al., 2013; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 2009; Kulier et al., 2012; Laibhen- 
Parkes, 2014; Saunders et al., 20i6Welch, Van Lunen, & Hankemeier, 2014). Of these, five studies 


30 


The Campbell Collaboration | www.campbellcollaboration.org 



also included the fifth key competency, evaluating the process of EBHC (Bradley et al., 2005; 
Fernandez et al., 2014; Ilic et al., 2015; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 
2014), while another four (Dizon et al., 2014; Ilic et al., 2013; Ramos-Morcillo et al., 2015; Schilling 
et al., 2006) included certain enabling competencies (epidemiology, biostatistics and basic 
searching skills). 
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Table 9.7: Intervention components 


Study ID 

Intervention 

Comparison 


E-learnin 

g components 

Face-to-face components 


Face-to-face components 

E-learning components 


Recorded PowerPoint 
presentations 

Online tutorial 

Online 

exercises / assignments / 
Clinical scenarios 

Access to databases 

Online support 
(tutor/feedback) 

Online tools (e.g. 
checklists, calculators) 

Asynchronous 
discussion list 

Online journal club 

Access to online site/ 
teaching material/ 
learning package 

Newsletter 

Mobile learning at 
bedside 

Clinical 

activities/assignments 

Access to clinical 
tutor/facilitator 

Didactic lectures 

Interactive workshop 

Small-group discussion 

Hands-on computer- 
based training 

No EBHC learning 

Didactic lectures 

Interactive workshop 

Small group discussions 

Hands-on computer- 
based training 

Journal club 

Recorded PowerPoint 
presentations 

Access to databases/ 
journals 

Teaching material 
received via email 

Access to facilitator on 
demand 

Access to online 
site/teaching material 

Pure e-learnin 

g vs no learning | 

Laibhen-Parkes 2014 

X 

















X 











Shilling 2006 

X 

X 




X 












X 











Welch 2014 









X 









X 











Blended learning vs no learning 1 

Bergold 2013 

X 


X 









X 

X 





X 











Dizon 2014 





X 

X 



X 





X 

X 



X 











Forsetlund 2003 




X 

X 


X 


X 

X 





X 



X 







X 




Kok 2013 









X 





X 

X 

X 

X 

X 











Ramos-Morcillo 2015 



X 


X 


X 


X 





X 




X 











Pure e-learning vs face-to-face learning | 

Bradley 2005 

X 



















X 









Davis 2007 

X 


















X 










Davis 2008 

X 


















X 










Hadley 2010 

X 


















X 










Horiuchi 2009 

X 


X 
















X 


X 








McLeod 2010 



X 




X 

X 















X 



X 



Blended learning vs face-to-face learning | 

Brettle 2013 


X 















X 


X 



X 







Ilic 2013 









X 



X 



X 

X 



X 

X 

X 








Ilic 2015 

X 








X 


X 



X 

X 

X 



X 

X 

X 








Kulier 2009 

X 


X 









X 

X 






X 

X 









Saunders 2016 


X 

X 


X 









X 





X 










Blended learning vs pure e-learning 


Fernandez 2014 (1) 

X 

X 







X 




















Fernandez 2014 (3) 





X 






X 

Fernandez 2014 (4) 


X 









X 

Kamin 2001 



X 




X 


X 












X 







X 

Kulier 2012 

X 











X 

X 











X 



X 


Pure e-learning vs pure e-learning | 

Brouwers 2011 (1) 


X 







X 

















X 



Brouwers 2011 (2) 


X 



X 




X 









Fernandez 2014 (2) 

X 

X 


























X 

MacRae 2004 







X 

X 

















X 
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Table 9.8: EBHC learning content of interventions 


Study ID 

Enabling 

competencies 

Key competencies 

Epidemiology 

Biostatistics 

Basic searching skills 

Critical thinking 

General EBHC 

Asking questions 

Accessing the 

literature 

Critical appraisal 

Applying the results 

Evaluating EBHC 

process 

Pure e-learning vs no learning 

Laibhen-Parkes 2014 






X 

X 

X 

X 


Shilling 2006 

X 


X 



X 

X 


X 


Welch 2014 






X 

X 

X 

X 

X 

Blended learning vs no learning 

Bergold 2013 






X 

X 

X 

X 


Dizon 2014 

X 




X 

X 

X 

X 

X 


Forsetlund 2003 





X 

X 

X 

X 

X 


Kok 2013 





X 

X 

X 

X 

X 


Ramos-Morcillo 2015 



X 


X 

X 

X 




Pure e-learning vs face-to-face learning 

Bradley 2005 






X 

X 

X 

X 

X 

Davis 2007 






X 

X 

X 

X 


Davis 2008 






X 

X 

X 

X 


Hadley 2010 






X 

X 

X 



Horiuchi 2009 





X 

X 

X 

X 

X 


Kamin 2001 






X 

X 




McLeod 2010 








X 



Blended learning vs face-to-face learning 

Brettle 2013 







X 




Ilic 2013 











flic 2015 





X 

X 

X 

X 

X 

X 

Kulier 2009 






X 

X 

X 

X 


Saunders 2016 





X 


X 

X 

X 

X 

Blended learning vs pure e-learning 

Fernandez 2014 





X 

X 

X 

X 

X 

X 

Kulier 2012 






X 

X 

X 

X 


Pure e-learning vs pure e-learning 

MacRae 2004 








X 



Brouwers 2011 








X 
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4-1-2.5 Duration of the interventions 

Duration of interventions in included studies ranged from a single, one hour session in two studies 
(Davis et al., 2007; Davis et al., 2008) to a journal club addressing one article per month and 
running over eight months in two studies (Macrae et al., 2004; McLeod et al., 2010). In one study, 
the intervention period was two weeks, consisting of five half day sessions (Bradley et al., 2005), 
while in another, the intervention consisted of two one-hour sessions which were one month apart 
(Brettle & Raynor, 2013). In nine studies, interventions were delivered over a three to eight week 
period, with learners progressing according to their own needs (Bergold et al., 2013; Brouwers et 
al., 2011; Hadley et al., 2010; Horiuchi et al., 2009; Kulier et al., 2009; Kulier et al., 2012; Laibhen- 
Parkes, 2014; Ramos-Morcillo et al., 2015; Saunders et al., 2016; Schilling et al., 2006; Welch, Van 
Lunen, & Hankemeier, 2014). One study only specified that the intervention was delivered during 
the “fall quarter” and that participants accessed the learning material according to their own needs 
(Kamin et al., 2001). In one study with three intervention arms, the interventions were delivered 
over a period of 15 weeks, with 10 hours per week allocated to studying the content. In one of the 
intervention arms, participants received three additional workshops of two hours each (Fernandez 
et al., 2014). The interventions in two studies comprised ten two-hour sessions delivered over two 
periods of two months each (Ilic et al., 2013; Ilic et al., 2015), while the intervention of another 
consisted of five contact days spread over a period of six months (Kok et al., 2013). Two studies 
with blended learning interventions included an initial workshop followed by a period of online 
support, during which participants engaged with the content according to their own needs. In one 
of these, (Dizon et al., 2014) the one day workshop consisted of six lectures and four practical 
sessions, while the period of online support lasted three months. In the other, the duration of the 
workshops varied from one to five days, while the entire intervention period lasted 18 months 
(Forsetlund et al., 2003). 

4. 1.2. 6 Learning theories 

In ten studies, the intervention was explicitly based on a learning theory or on one or more learning 
approaches. Three studies referred to distributed learning (Bergold et al., 2013; Bradley et al., 

2005; Horiuchi et al., 2009), another three based their intervention on the adult learning theory 
(Dizon et al., 2014; Macrae et al., 2004; Schilling et al., 2006), and two interventions made use of a 
learner centred approach (Bergold et al., 2013; Kulier et al., 2012). Other theories and learning 
approaches included the SPICES framework (Bergold et al., 2013), the constructivist model of 
learning (Fernandez et al., 2014), the innovation-diffusion process (Forsetlund et al., 2003), novice 
to expert theory and theory of planned behaviour (Laibhen-Parkes, 2014), just-in-time learning 
(Kulier et al., 2012), and the utility of reflective learning and collaborative learning (Schilling et al., 
2006). 

4.1. 2 . 7 Comparisons 

Comparisons included no EBHC learning, face-to-face learning of EBHC and e-learning of EBHC. 
Three studies compared pure e-learning to no learning (Laibhen-Parkes, 2014; Schilling et al., 
2006; Welch, Van Lunen, & Hankemeier, 2014), while five studies compared blended learning to 
no learning (Bergold et al., 2013; Dizon et al., 2014; Forsetlund et al., 2003; Kok et al., 2013; 
Ramos-Morcillo et al., 2015). Seven studies compared pure e-learning to face-to-face learning 
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(Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010; Horiuchi et al., 
2009; Kamin et al., 2001; McLeod et al., 2010) and five studies compared blended learning to face- 
to-face learning (Brettle & Raynor, 2013; Ilic et al., 2013; Ilic et al., 2015; Kulier et al., 2009; 
Saunders et al., 2016). Three studies compared blended learning to pure e-learning (Fernandez et 
al., 2014; Kamin et al., 2001; Kulier et al., 2012) and two studies compared pure e-learning to other 
purely e-learning interventions (Brouwers et al., 2011; Macrae et al., 2004). 

Amongst the sixteen trials that compared e-learning interventions to either face-to-face, blended or 
other e-learning interventions, there were nine single-component (Bradley et al., 2005; Brettle & 
Raynor, 2013; Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; Fernandez et al., 2014; 
Hadley et al., 2010; Macrae et al., 2004; Saunders et al., 2016) and nine multi-faceted (Fernandez 
et al., 2014; Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 2015; Kamin et al., 2001; Kulier et al., 
2009; Kulier et al., 2012; McLeod et al., 2010) comparisons. Comparisons of all included studies 
were delivered via stand-alone teaching as opposed to integrated teaching. Four of the control 
interventions involved collaborative learning (Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 

2015; Kamin et al., 2001), while the rest relied on individual learning. 

Amongst the face-to-face and blended learning interventions, didactic lectures were the most 
common intervention component (Brettle & Raynor, 2013; Davis et al., 2007; Davis et al., 2008; 
Fernandez et al., 2014; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 2015; 
Kulier et al., 2009; Saunders et al., 2016). For four of these (Davis et al., 2007; Davis et al., 2008; 
Hadley et al., 2010; Saunders et al., 2016), it was the only intervention component. Other face-to- 
face components were small group discussions (Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 
2015; Kamin et al., 2001), an interactive workshop (Bradley et al., 2005; Ilic et al., 2015; Kulier et 
al., 2009), hands-on computer training (Brettle & Raynor, 2013; Fernandez et al., 2014) and a 
journal club (McLeod et al., 2010). The most common e-learning component of blended 
comparisons was access to the online teaching site and materials (Fernandez et al., 2014; Kamin et 
al., 2001). Other components included receiving teaching material via email (Brouwers et al., 2011; 
McLeod et al., 2010), access to databases and journals (Forsetlund et al., 2003; Macrae et al., 

2004) access to a tutor on demand (Kulier et al., 2012) and recorded PowerPoint presentations 
(Kulier et al., 2012). 

The content and duration of the control interventions was the same in both intervention and 
comparison groups. 

4.1.2.8 Outcomes 

Primary outcomes 

Twelve trials reported on EBHC knowledge (Bergold et al., 2013; Bradley et al., 2005; Davis et al., 
2007; Davis et al., 2008; Fernandez et al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; 
Horiuchi et al., 2009; Kamin et al., 2001; Kulier et al., 2009; Kulier et al., 2012; Welch, Van Lunen, 
& Hankemeier, 2014). Six trials reported on EBHC knowledge and skills as a combined outcome 
(Dizon et al., 2014; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 2013; Laibhen-Parkes, 2014; Ramos- 
Morcillo et al., 2015). Across studies, knowledge was measured with various validated tools or 
adaptations of these. The most commonly used tools were the Berlin questionnaire and the Fresno 
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test. Four studies used a questionnaire based on both the Fresno test and the Berlin questionnaire 
(Davis et al., 2007; Davis et al., 2008; Kulier et al., 2009; Kulier et al., 2012), four studies used an 
adapted version of the Fresno test (Bergold et al., 2013; Dizon et al., 2014; Kok et al., 2013; 
Laibhen-Parkes, 2014), and two studies made use of the Berlin questionnaire or adaptations 
thereof (Ilic et al., 2013; Ilic et al., 2015). Both these studies made use of additional assessments to 
measure EBHC knowledge, (Ilic et al., 2013) made use of two assignment tasks, while (Ilic et al., 
2015) used the validated ACE tool. Two studies (Bradley et al., 2005; Hadley et al., 2010) made use 
of a questionnaire validated by Taylor et al. (Taylor et al., 2001). Ramos-Morcillo et al. (2015) used 
a Spanish version of the Evidence-based Practice Questionnaire (EBPQ) and Saunders et al., (2016) 
measured EBHC knowledge as part of the Evidence-based Readiness Inventory (ERI). Two studies 
measured knowledge with questionnaires developed by the study authors or working group (Kamin 
et al., 2001; Welch, Van Lunen, & Hankemeier, 2014), another (Forsetlund et al., 2003) measured 
self-reported knowledge on a Likert scale. In a further study knowledge scores were based on 
assignment marks (Fernandez et al., 2014). 

Eight studies reported on EBHC skills (Bradley et al., 2005; Brettle & Raynor, 2013; Brouwers et 
al., 2011; Fernandez et al., 2014; Kulier et al., 2012; Macrae et al., 2004; McLeod et al., 2010; 
Schilling et al., 2006). In four studies participants had to critically appraise an article and were 
given scores for their appraisals (Bradley et al., 2005; Brouwers et al., 2011; Macrae et al., 2004; 
McLeod et al., 2010). Two studies scored the searches of participants (Brettle & Raynor, 2013; 
Schilling et al., 2006) and one study used an objective, structured clinical examination to measure 
EBHC skills (Kulier et al., 2012). 

Thirteen studies reported attitude towards EBHC (Bradley et al., 2005; Brouwers et al., 2011; Davis 
et al., 2007; Davis et al., 2008; Dizon et al., 2014; Forsetlund et al., 2003; Ilic et al., 2013; Ilic et al., 
2015; Kulier et al., 2009; Kulier et al., 2012; Laibhen-Parkes, 2014; Ramos-Morcillo et al., 2015; 
Schilling et al., 2006). All the studies made use of Likert scale questionnaires. 

Five trials evaluated EBHC behaviour (Dizon et al., 2014; Forsetlund et al., 2003; Ilic et al., 2015; 
Kok et al., 2013; Ramos-Morcillo et al., 2015), of which two (Ilic et al., 2015; Ramos-Morcillo et al., 
2015) measured behaviour with the Evidence-based Practice Questionnaire (EBPQ), requiring 
participants to rate their EBHC behaviour using a 7-point Likert scale. 

Dizon et al. (2014) made use of activity diaries. EBHC behaviour of physiotherapists measured 
through activity diaries was categorized into evidence-based practice (EBP) behaviours 
(formulating PICO questions, logging PICO questions, searching research for evidence, appraising 
evidence and applying evidence) and non-EBP behaviours (asking colleagues, asking medical 
doctors and reading textbooks). This was reported for new or unique cases as well as for usual 
cases. 

One study (Forsetlund et al., 2003) analysed the content of the local health service reports and 
asked participants to complete a hypothetical assignment and a questionnaire on self-reported 
behaviour. Scores (1-5) for the assignments were based on the extent to which the document 
reflected EBHC elements. 


36 


The Campbell Collaboration | www.campbellcollaboration.org 



Kok et al. (2013) assessed the frequency of evidence of sufficient quality in disability evaluation 
reports through six indicators - presence of evidence; a discernible EBM question; search strategy; 
EBM source; and evaluation of quality. 

Eight studies measured outcomes after the intervention only (Bradley et al., 2005; Brouwers et al., 
2011; Fernandez et al., 2014; Ilic et al., 2013; Ilic et al., 2015; Laibhen-Parkes, 2014; Macrae et al., 
2004; McLeod et al., 2010), while all other studies measured outcomes both before and after the 
intervention. 

Secondary outcomes 

Satisfaction with learning was reported in four studies (Bergold et al., 2013; Brouwers et al., 2011; 
Horiuchi et al., 2009; Ilic et al., 2015) and enablers and barriers of EBHC learning in one study 
(Ilic et al., 2015). Attrition of learners was reported in 17 trials (Bergold et al., 2013; Bradley et al., 
2005; Brettle & Raynor, 2013; Brouwers et al., 2011; Davis et al., 2008; Dizon et al., 2014; Hadley 
et al., 2010; Horiuchi et al., 2009; Ilic et al., 2013; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 
2009; Kulier et al., 2012; Laibhen-Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; 
Saunders et al., 2016). 

4. 1.2 . 9 Educational context 

Educational context in terms of learner context, institutional context and socio-economic context 
was not explicitly addressed, although for some studies, having access to a computer and internet 
was a pre-requisite to participating in the study. 

4.1.3 Ongoing studies 

We identified one study (Schneider et al., 2014) that is still on-going (Table Q.4) . 

4.1.4 Excluded studies 

From those studies subjected to full-text screening we excluded 30 studies. Reasons for exclusion 
are listed in the table of excluded studies (Table p..O . 


4.2 RISK OF BIAS IN INCLUDED STUDIES 


Overall, we judged risk of bias to be moderate. Details of our judgement of risk of bias are 
presented as part of the characteristics of included studies (Table 9 . 3 ) . The risk of bias across 
studies is summarised in Figure 5 and Figure 6 . 

4.2.1 Selection bias 

In 17 RCTs the allocation sequence was adequately generated (Bergold et al., 2013; Bradley et al., 
2005; Brettle & Raynor, 2013; Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; Dizon et 
al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2015; Kok et 
al., 2013; Kulier et al., 2009; Kulier et al., 2012; Laibhen-Parkes, 2014; Saunders et al., 2016; 
Welch, Van Lunen, & Hankemeier, 2014). Four non-RCTs did not make use of randomisation 
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(Fernandez et al., 2014; Ilic et al., 2013; Ramos-Morcillo et al., 2015; Schilling et al., 2006) and 
three studies did not describe how the allocation sequence was generated so they were judged as 
having unclear risk of selection bias (Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 2010). 
Allocation concealment was adequate in eight RCTs (Bergold et al., 2013; Bradley et al., 2005; 
Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Kok et al., 2013; Kulier et al., 2009; Kulier 
et al., 2012), not done in the four non-RCTs (Fernandez et al., 2014; Ilic et al., 2013; Ramos- 
Morcillo et al., 2015; Schilling et al., 2006) and not described and thus judged as unclear in 12 
studies (Brettle & Raynor, 2013; Brouwers et al., 2011; Forsetlund et al., 2003; Hadley et al., 2010; 
Kamin et al., 2001; Laibhen-Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; Saunders et 
al., 2016; Welch, Van Lunen, & Hankemeier, 2014). 

Baseline outcome measurements were similar between groups in 10 trials (Bergold et al., 2013; 
Brettle & Raynor, 2013; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; Kok et al., 2013; 
Kulier et al., 2009; Kulier et al., 2012; Ramos-Morcillo et al., 2015; Welch, Van Lunen, & 
Hankemeier, 2014) and not adequately reported in 13 studies (Bradley et al., 2005; Brouwers et al., 
2011; Fernandez et al., 2014; Forsetlund et al., 2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic 
et al., 2013; Ilic et al., 2015; Kamin et al., 2001; Macrae et al., 2004; McLeod et al., 2010; Saunders 
et al., 2016; Schilling et al., 2006). In one study (Laibhen-Parkes, 2014) there was a significant 
difference in EBHC attitudes between groups at baseline. 

Baseline characteristics were similar between groups in six studies (Bradley et al., 2005; Kok et al., 
2013; Kulier et al., 2012; Laibhen-Parkes, 2014; Ramos-Morcillo et al., 2015; Schilling et al., 2006), 
not similar in four studies (Brouwers et al., 2011; Davis et al., 2007; Forsetlund et al., 2003; 
Saunders et al., 2016), not measured in three studies (Ilic et al., 2013; Ilic et al., 2015; Kulier et al., 
2009) and unclear in 11 studies (Bergold et al., 2013; Brettle & Raynor, 2013; Davis et al., 2008; 
Dizon et al., 2014; Fernandez et al., 2014; Hadley et al., 2010; Horiuchi et al., 2009; Kamin et al., 
2001; Macrae et al., 2004; McLeod et al., 2010; Welch, Van Lunen, & Hankemeier, 2014). 

4.2.2 Attrition bias 

Fourteen studies (Brouwers et al., 2011; Davis et al., 2008; Dizon et al., 2014; Forsetlund et al., 
2003; Hadley et al., 2010; Horiuchi et al., 2009; Ilic et al., 2015; Kok et al., 2013; Kulier et al., 
2009; Laibhen-Parkes, 2014; Macrae et al., 2004; McLeod et al., 2010; Saunders et al., 2016; 
Welch, Van Lunen, & Hankemeier, 2014) were judged as having high risk of attrition bias, due to 
significant loss to follow up. Five studies (Fernandez et al., 2014; Kamin et al., 2001; Kulier et al., 
2012; Ramos-Morcillo et al., 2015; Schilling et al., 2006) did not adequately describe the flow of all 
participants and were judged as having unclear risk of attrition bias, whereas five studies (Bergold 
et al., 2013; Bradley et al., 2005; Brettle & Raynor, 2013; Davis et al., 2007; Ilic et al., 2013) were 
judged as having low risk of attrition bias. 

4.2.3 Detection bias 

Eleven studies (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Dizon et al., 2014; 
Forsetlund et al., 2003; Ilic et al., 2015; Kamin et al., 2001; Kok et al., 2013; Laibhen-Parkes, 2014; 
Macrae et al., 2004; Schilling et al., 2006) adequately blinded outcome assessors and were judged 
as having low risk of detection bias. We judged three studies (Fernandez et al., 2014; Ilic et al., 
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2013; McLeod et al., 2010) as having high risk of detection bias, since outcome assessors were not 
adequately blinded. The remaining 10 studies (Bergold et al., 2013; Brettle & Raynor, 2013; 
Brouwers et al., 2011; Hadley et al., 2010; Horiuchi et al., 2009; Kulier et al., 2009; Kulier et al., 
2012; Ramos-Morcillo et al., 2015; Saunders et al., 2016; Welch, Van Lunen, & Hankemeier, 2014) 
did not adequately report on blinding and were judged as having unclear risk of detection bias. 

4.2.4 Contamination 

Eight trials (Brouwers et al., 2011; Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010; Ilic et 
al., 2015; Kulier et al., 2009; Kulier et al., 2012; McLeod et al., 2010) were adequately protected 
against contamination. We judged two trials as having high risk of contamination (Bergold et al., 
2013; Saunders et al., 2016;); in the remaining 14 trials (Bradley et al., 2005; Brettle & Raynor, 
2013; Dizon et al., 2014; Fernandez et al., 2014; Forsetlund et al., 2003; Horiuchi et al., 2009; 
Kamin et al., 2001; Kok et al., 2013; Laibhen-Parkes, 2014; Macrae et al., 2004; Schilling et al., 
2006; Welch, Van Lunen, & Hankemeier, 2014) the risk of contamination was unclear. 

4.2.5 Reporting biases 

We judged one study (Kulier et al., 2012) as having unclear risk of reporting bias, since the authors 
only reported the baseline measurements for one of the outcomes. The remaining studies all 
reported on pre-specified outcomes and were judged as having low risk of reporting bias. 

4.2.6 Other bias 

We judged five studies (Bergold et al., 2013; Fernandez et al., 2014; Kamin et al., 2001; Kulier et 
al., 2012; Laibhen-Parkes, 2014) as having unclear risk of other bias and the remaining studies as 
low risk of other bias. 

4.2.7 Additional risk of bias in cluster RCT s 

4.2. 7.1 Recruitment bias 

Three cluster randomised trials (Hadley et al., 2010; Kok et al., 2013; McLeod et al., 2010) had low 
risk of recruitment bias, since participants were recruited before randomisation of clusters. For 
three studies (Ilic et al., 2015; Kulier et al., 2009; Laibhen-Parkes, 2014) it was unclear whether 
participants were recruited before or after randomisation of clusters, and one trial (Kulier et al., 
2012) was at high risk of recruitment bias, since participants were recruited after randomisation. 

4.2. 7.2 Baseline imbalance 

In three studies (Kok et al., 2013; Kulier et al., 2012; Laibhen-Parkes, 2014), baseline 
characteristics were not significantly different between groups whereas in two other studies 
(Hadley et al., 2010; McLeod et al., 2010), it was unclear whether there were baseline imbalances 
between groups. Two studies (Ilic et al., 2015; Kulier et al., 2009) did not measure baseline 
characteristics and were judged as having high risk of baseline imbalances. 
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4.2. 7*3 Loss of clusters 

One study (Kulier et al., 2012) lost more than 20% of clusters and was judged as having high risk of 
bias. Five studies (Hadley et al., 2010; Ilic et al., 2015; Kok et al., 2013; Laibhen-Parkes, 2014; 
McLeod et al., 2010) had unclear risk of bias, since loss of clusters was not reported, and one study 
(Kulier et al., 2009) had low risk of bias. 

4.2. 7.4 Incorrect analysis 

We judged five studies as having low risk of bias (Hadley et al., 2010; Kok et al., 2013; Kulier et al., 
2009; Kulier et al., 2012; McLeod et al., 2010), since authors reported that adjustment for 
clustering was done. Two studies (Ilic et al., 2015; Laibhen-Parkes, 2014) did not adjust results for 
clustering and we judged them as having high risk of bias. 

4.2.7.5 Compatibility with individually randomised controlled trials 

Results from five cluster RCTs were compatible with the individually randomised trials and had 
low risk of bias. Two studies that did not adjust results for clustering were judged as not being 
compatible with individually randomised studies (Ilic et al., 2015; Laibhen-Parkes, 2014). 


4.3 SYNTHESIS OF RESULTS 


Twenty-three studies were included in the synthesis of results. One study (Bergold et al., 2013) was 
excluded from the synthesis because the timing of measurement of outcomes differed for the 
intervention and comparison group. EBHC knowledge and skills scores were measured at baseline 
and after the intervention in the intervention group. For the waitlist-control group, however, 
baseline scores were only measured at three months. 

We pooled results using random-effects meta-analysis and report on the pooled standardised mean 
difference for each outcome. A summary of the results is presented in Table 9.9. Interventions 
differed considerably between studies, so the overall mean effect may not be an accurate reflection 
of the impacts for any particular intervention type. 

4.3.1 Pure e-learning vs no learning 

One non-randomised trial (Schilling et al., 2006) and two RCTs (Laibhen-Parkes, 2014; Welch et 
al., 2014) compared pure e-learning to no learning (Table 12.1I . Overall, results favoured pure e- 
learning interventions. 

4.3.1. 1 Primary outcomes 

4.3.1 .1 .1 EBHC knowledge 

One RCT (Welch et al., 2014) reported on EBHC knowledge (Analysis 1.1I . EBHC knowledge scores 
were higher for participants in the group that received pure e-learning compared to the group that 
received no learning (SMD 0.71 95%CI 0.40 to 1.01; 1 study, n=i75). The SMD of 0.71 can be 
interpreted as a medium effect size. 


40 


The Campbell Collaboration | www.campbellcollaboration.org 


4-3-1 .1 .2 EBHC knowledge and skills 


One RCT (Laibhen-Parkes, 2014) reported on EBHC knowledge and skills. There was no significant 
difference between groups in post-intervention knowledge and skills scores (SMD 0.47 95%CI - 
0.27 to 1.21, 1 study, n=29) (Analysis 1.2I . 

4.3.1 .1 .3 EBHC skills 

One non-RCT (Schilling et al., 2006) reported on EBHC skills. The mean MEDLINE searching 
score for the intervention group was 2.9 (11=74) compared to 2.1 (n=s8) in the control group 
(p<0.05). The authors did not report standard deviations (SD) or 95%CI for the means. 

Participants in the intervention group had greater odds of calculating the number needed to treat 
(NNT) correctly, compared to the control group (OR 5.4 95%CI 2.7 to 11.0, 11=179). 

4.3. 1.1. 4 EBHC attitude 

One non-RCT (Schilling et al., 2006) and one RCT (Laibhen-Parkes, 2014) reported on EBHC 
attitude. Post-intervention attitude scores, adjusted for pre-intervention scores, were higher for the 
intervention group compared to the control group (SMD 1.05 95% Cl 0.26 to 1.83; 1 study, n=29) 
(Analysis 1.3) . The SMD of 1.05 can be interpreted as a large effect size. 

In Schilling et al. (2006), the MD in change of attitude scores from baseline to post-intervention, 
was significantly higher (p<o.os) in the intervention group for seven of the eight statements 
related to EBHC attitude. 

4.3. 1.1. 5 EBHC behaviour 

None of the studies comparing EBHC pure e-learning to no intervention reported on EBHC 
behaviour outcomes. 

4.3.1.2 Secondary outcomes 

4.3.1.2.1 Satisfaction of students with method of learning 
No studies reported on this outcome. 

4.3. 1.2. 2 Satisfaction of educator with method of learning 
No studies reported on this outcome. 

4.3.1.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3. 1.2. 4 Barriers to the method of learning EBHC 
No studies reported on this outcome. 

4.3.1 .2.5 Cost 


41 


The Campbell Collaboration | www.campbellcollaboration.org 


No studies reported on this outcome. 


4.3. 1.2. 6 Attrition of learners 

Two studies reported on attrition rates (Laibhen-Parkes, 2014; Welch, Van Lunen, & Hankemeier, 
2014). There was no difference in attrition rates between the intervention and control group for 
this comparison (RR 1.03; 95% Cl 0.9 to 1.18; Chi 2 =i.58, P=o.2i; I 2 =37%; n=53i) fAnalvsis 1.4k 

4.3. 1. 2. y Learner adherence 

No studies reported on this outcome. 

4.3.1. 2. 8 Evidence-based practice 
No studies reported on this outcome. 

4.3. 1.2. 9 Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.1.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.1.2.11 Individual health outcomes 
No studies reported on this outcome. 

4.3.1.2.12 Population health outcomes 
No studies reported on this outcome. 

4.3.2 Blended learning vs no learning 

One non-RCT (Ramos-Morcillo et al., 2015) and four RCTs (Bergold et al., 2013; Dizon et al., 2014; 
Forsetlund et al., 2003; Kok et al., 2013) compared blended learning with no learning (Table 12.2k 
Overall, results favoured blended learning interventions for EBHC knowledge and skills. 

4.3.2.1 Primary outcomes 

4.3. 2. 1.1 EBHC knowledge 

One RCT (Forsetlund et al., 2003) reported on EBHC knowledge. Knowledge scores were higher in 
the blended learning group compared to the control group (SMD 0.50 95%CI 0.13 to 0.86; 1 study, 
n=ii9) (Analysis 2.1k The SMD of 0.5 can be interpreted as a medium effect size. 

4.3. 2.1 .2 EBHC knowledge and skills 

One non-RCT (Ramos-Morcillo et al., 2015) and two RCTs (Dizon et al., 2014; Kok et al., 2013) 
reported on EBHC knowledge and skills. These trials measured scores at multiple time points, i.e. 
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directly post-intervention as well as one month (Kok et al., 2013; Ramos-Morcillo et al., 2015), 3 
months (Dizon et al., 2014) and 6 months (Kok et al., 2013) after the intervention. We report on 
the short term as well as long term EBHC knowledge and skills scores separately. 

For EBHC knowledge and skills scores measured immediately post-intervention we included two 
trials (Dizon et al., 2014; Ramos-Morcillo et al., 2015) in the random-effects meta-analysis 
(Analysis 2.2I . Although results from both trials favoured blended learning, the pooled effect 
showed no difference in EBHC knowledge and skills scores (SMD 1.40 95%CI -0.06 to 2.85, 2 
studies, n=i63). Heterogeneity was high (Tau 2 =i.02; Chi 2 =i3.8o, P=o.oo02; I 2 =93%) and the test 
for subgroup differences was significant (p=0.0002; I 2 =92%). The SMD of 1.4 can be interpreted as 
a very large effect size, although the results are not statistically significant. 

For EBHC knowledge and skills scores measured one month after the intervention, we included 
two trials (Kok et al., 2013; Ramos-Morcillo et al., 2015) in the random-effects meta-analysis 
(Analysis 2.3) . EBHC knowledge and skills scores were higher amongst participants in the blended 
learning group, compared to the group that received no learning (SMD 0.9 95%CI 0.42 to 1.38, 2 
studies, 11=141). Heterogeneity was high (Tau 2 = 0.08; Chi 2 = 3.25, P = 0.07; I 2 = 69%) with a 
significant test for subgroup differences (p=0.07; I 2 =69.2%). The SMD of 0.9 can be interpreted as 
a large effect size. 

For EBHC knowledge and skills scores measured more than 3 months after the intervention, we 
included two trials (Dizon et al., 2014; Kok et al., 2013) in the random-effects meta-analysis 
(Analysis 2.4) . EBHC knowledge and skills scores remained higher in the blended learning group 
compared to the group that received no learning (SMD 1.11 95%CI 0.80 to 1.42, 2 studies, n=i86). 
Heterogeneity was high (Chi 2 = 5.33, P = 0.02; I 2 = 81%). The SMD of 1.11 can be interpreted as a 
large effect size. 

4. 3. 2.1 .3 EBHC skills 

No studies reported on EBHC skills. 

4.3. 2. 1.4 EBHC attitude 

One non-RCT (Ramos-Morcillo et al., 2015) and three RCTs (Dizon et al., 2014; Forsetlund et al., 
2003; Kok et al., 2013; Ramos-Morcillo et al., 2015) reported on post-intervention attitude scores. 
Three trials measured scores at multiple time points i.e. directly post-intervention as well as one 
month (Kok et al., 2013; Ramos-Morcillo et al., 2015), 3 months (Dizon et al., 2014) and 6 months 
(Kok et al., 2013) after the intervention. 

For EBHC attitudes measured directly post-intervention we included one non-RCT (Ramos- 
Morcillo et al., 2015) and one RCT (Forsetlund et al., 2003) in the random-effects meta-analysis 
(Analysis 2.5) . There was no difference in EBHC attitude scores between groups (SMD 0.17 95%CI 
-0.09 to 0.48, 2 studies, 11=226). Heterogeneity was absent (Tau 2 = 0.00; Chi 2 = 0.15, P = 0.70; I 2 
= 0%). 


For EBHC attitude scores measured one month post-intervention, we included one non-RCT 
(Ramos-Morcillo et al., 2015) and one RCT (Kok et al., 2013) in the random-effects meta-analysis 
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( Analysis 2.6 ). There was no difference in EBHC attitude scores between groups (SMD 0.05 95%CI 
-0.34 to 0.44). Heterogeneity was moderate but non-significant (Tau 2 = 0.05; Chi 2 = 2.36, P = 
0.12; I 2 = 58%). 

One RCT (Kok et al., 2013) measured EBHC attitude six months post-intervention and found no 
difference between groups (SMD 0.32 95%CI -0.02 to 0.67, 1 study, n=i32) (Analysis 2.7b 

In Dizon et al. (2014), attitude scores (3-point Likert scale) in the blended learning group were 
significantly different compared to the wait-list control group for two of the six attitude statements 
(“I would lack confidence in undertaking a literature search” and “I would feel confident in 
undertaking a critical appraisal”) immediately post-intervention and three months post 
intervention. For the statement “I would find it difficult to change what I already do in clinical 
practice”, there was a significant difference between groups immediately post-intervention, but not 
at three months post-intervention. 

4. 3. 2. 1.5 EBHC behaviour 

One non-RCT (Ramos-Morcillo et al., 2015) and three RCTs (Dizon et al., 2014; Forsetlund et al., 
2003; Kok et al., 2013) reported on EBHC behaviour. Dizon et al. (2014) and Kok et al. (2013) 
measured behaviour 3 months after the intervention. Forsetlund et al. (2003) only measured 
behaviour directly after the intervention, while Ramos-Morcillo et al. (2015) measured EBHC 
behaviour directly after, as well as 40 days after the intervention. 

For EBHC behaviour measured directly post intervention, we included one non-RCT (Ramos- 
Morcillo et al., 2015) and one RCT (Forsetlund et al., 2003) in the random-effects meta-analysis 
(Analysis 2.8) . Overall, there was no difference between groups (SMD 0.06 95%CI -0.28 to 0.40, 2 
studies, n=207). Heterogeneity was low (Tau 2 = 0.02; Chi 2 = 1.53, P = 0.22; I 2 = 35%) and there 
was no difference between subgroups (p=0.22, I 2 =34-8%). 

For EBHC behaviour measured one month post intervention, we included one non-RCT (Ramos- 
Morcillo et al., 2015) in the random-effects meta-analysis (Analysis 2.q) . There was no significant 
difference between groups (SMD 0.19 95%CI -0.19 to 0.56, 1 study, n=i09). 

For EBHC behaviour measured 3+ months post-intervention, we included one RCT (Kok et al., 
2013) in the random-effects meta-analysis (Analysis 2.10) . Behaviour scores were greater for the 
blended learning compared to no learning results (SMD 0.61 95%CI 0.21 to 1.01, 1 study, n=ioo). 
The SMD can be interpreted as a medium effect size. 

In Dizon et al. (2014), 18 participants in the intervention and 19 in the wait-list control group 
submitted activity diaries three months post-intervention. There was a significant difference in 
EBHC behaviour between groups for all categories when faced with a new or unique case (n=24), 
and significant differences for all but one category (logging PICO questions) when participants 
were faced with usual cases (n=i9). For non-EBP behaviour, there was a significant difference 
between groups for two (asking medical doctors and reading textbooks) of the three categories 
when faced with a new case, but no significant difference between groups when faced with a usual 
case. 
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4'3' 2 ' 2 Secondary outcomes 

4.3.2.2.1 Satisfaction of students with method of learning 
No studies reported on this outcome. 

4-3-2.2.2 Satisfaction of educator with method of learning 
No studies reported on this outcome. 

4.3.2.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3.2.2.4 Barriers to the method of learning EBHC 
No studies reported on this outcome. 

4.3.2.2.5 Cost 

No studies reported on this outcome. 

4-3.2.2.6 Attrition of learners 

Three studies (Bergold et al., 2013; Dizon et al., 2014; Kok et al., 2013) reported on attrition rates. 
Random-effects meta-analysis fAnalvsis 2.11) showed no significant difference in attrition of 
learners between groups (RR 0.82 95%CI 0.55 to 1.20; n=306; Chi 2 =0.75, P=o.69; I 2 =o%). 

4.3.2. 2 . 7 Learner adherence 

No studies reported on this outcome. 

4.3.2. 2. 8 Evidence-based practice 
No studies reported on this outcome. 

4.3. 2. 2. g Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.2.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.2.2.11 Individual health outcomes 
No studies reported on this outcome. 

4.3.2.2.12 Population health outcomes 
No studies reported on this outcome. 
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4.3*3 Pure e-learning vs face-to-face learning 

Six RCTs (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; Hadley et al., 2010; Horiuchi 
et al., 2009; McLeod et al., 2010) compared pure e-learning to face-to-face learning (Table 12.3) . 
Overall, there was no difference between groups for any of the outcomes. 

4.3.3.1 Primary outcomes 

4.3. 3. 1.1 EBHC knowledge 

Five RCTs reported on EBHC knowledge (Bradley et al., 2005; Davis et al., 2007; Davis et al., 

2008; Hadley et al., 2010; Horiuchi et al., 2009) and were included in the random-effects meta- 
analysis (Analysis 3.1) . There was no difference in knowledge scores between pure e-learning 
compared to face-to-face learning (SMD -0.03 95%CI -0.26 to 0.20; 5 studies, n=632). 
Heterogeneity was moderate (Tau 2 = 0.03; Chi 2 = 7.74, P = 0.10; I 2 = 48%) and caused by one 
outlier (Davis et al., 2008). A possible reason for Davis et al. (2008) being the only study where 
results favoured face-to-face learning is the attrition rate amongst participants in the pure e- 
learning group (44/114; 39%) compared to the face-to-face learning group (6/115; 5 %)- When 
removing this trial from the analysis, the effect does not change (SMD 0.05 95%CI -0.14 to 0.25) 
and heterogeneity is reduced (Tau 2 = 0.00; Chi 2 = 3.24, P = 0.36; I 2 = 7%). 

4.3. 3. 1.2 EBHC knowledge and skills 
No studies reported on this outcome. 

4.3.3.1.3 EBHC skills 

Two RCTs reported on EBHC skills (Bradley et al., 2005; McLeod et al., 2010) and were included in 
the random-effects meta-analysis (Analysis 3.2k There was no difference in EBHC skills scores 
between groups (SMD -0.15 95%CI -0.34 to 0.04; 2 studies, n=457), with no significant 
heterogeneity (Tau 2 = 0.00; Chi 2 = 1.04, P = 0.31; I 2 = 4%). 

4.3.3.1.4 EBHC attitude 

Three RCTs (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008) reported on EBHC attitude. 
Data from one RCT (P. Bradley et al., 2005) showed no difference in EBHC attitude scores between 
groups (SMD o.ll 95%CI -0.27 to 0.48, 1 study, 11=111) (Analysis 3 . 3 ) . Change in EBHC attitude 
from baseline to post-intervention was similar between groups in Davis et al. (2007) and Davis et 
al. (2008). Authors did not report on means and SDs. 

4-3-3 -1-5 EBHC behaviour 

No studies reported on this outcome. 

4.3.3.2 Secondary outcomes 

4.3.3.2.1 Satisfaction of students with method of learning 
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One study (Horiuchi et al., 2009) explored learners’ satisfaction with the learning modality and 
reported selected quotes from both groups. The intervention group felt that learning at their own 
pace and in their own time was very convenient, but some also felt that they at times lacked 
motivation to engage with the content. The face-to-face learning group enjoyed the interaction and 
small group discussions during the sessions but some could not fit the sessions into their schedule. 

4.3.3.2.2 Satisfaction of educator with method of learning 
No studies reported on this outcome. 

4. 3. 3.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3.3.2.4 Barriers to the method of learning EBHC 
No studies reported on this outcome. 

4 - 3 - 3 - 2.5 Cost 

No studies reported on this outcome. 

4.3.3.2.6 Attrition of learners 

Six studies reported on attrition rates (Bradley et al., 2005; Davis et al., 2007; Davis et al., 2008; 
Hadley et al., 2010; Horiuchi et al., 2009; McLeod et al., 2010). In Davis et al. (2007), no 
participants were lost to follow-up. 

Random-effects meta-analysis (Analysis 3.4) of the attrition rates showed no difference for 
attrition rates between e-learning and face-to-face learning groups (RR 1.24 95% Cl 0.59 to 2.59; 5 
studies n=ll75; Tau 2 = 0.55; Chi 2 = 35.20, P<0. 00001); I 2 = 89%). 

4 .3.3.2.y Learner adherence 

No studies reported on this outcome. 

4.3. 3. 2. 8 Evidence-based practice 
No studies reported on this outcome. 

4. 3. 3.2.9 Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.3.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.3.2.11 Individual health outcomes 
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No studies reported on this outcome. 

4.3.3.2.12 Population health outcomes 
No studies reported on this outcome. 

4.3.4 Blended learning vs face-to-face learning 

One non-RCT (Ilic et al., 2013) and four RCTs (Brettle & Raynor, 2013; Ilic et al., 2015; Kulier et 
al., 2009; Saunders et al., 2016) compared blended learning to face-to-face learning (Table 12.4) . 
Overall, results were similar between groups for EBHC knowledge and skills, but favoured the 
blended learning approach for EBHC attitude and behaviour. 

4.3.4.1 Primary outcomes 

4.3. 4.1.1 EBHC knowledge 

Two RCTs reported on EBHC knowledge (Kulier et al., 2009; Saunders et al., 2016) and were 
included in the random-effects meta-analysis (Analysis 4.1) . Results from Kulier et al. (2009) 
showed no difference in EBHC knowledge scores between groups (SMD 0.28 9596CI-0.23 to 0.79; 1 
study, n=6i). 

Saunders et al. (2016) did not provide means and SDs and we were thus not able to include these in 
the meta-analysis. We contacted authors and are awaiting their response. 

4.3. 4.1. 2 EBHC knowledge and skills 

One non-RCT (Ilic et al., 2013) and one RCT (Ilic et al., 2015) reported on EBHC knowledge and 
skills and were included in the random-effects meta-analysis (Analysis 4.2~) . There was no 
difference in EBHC knowledge and skills scores between groups (SMD -0.23 95%CI -0.52 to 0.06; 

2 studies; n=208). Heterogeneity was low (Tau 2 = 0.00; Chi 2 = 0.01, P = 0.93; I 2 = 0%) and there 
was no significant difference between subgroups (p=0.93, 1 2 =o%). 

4.3. 4.1.3 EBHC skills 

One RCT reported on EBHC skills (Brettle & Raynor, 2013). Post-intervention searching scores 
were not significantly different between groups after the first session (SMD -0.21 95%CI -0.68 to 
0.26; n=70) (Analysis 4 . 2 k 

4.3.4.1.4 EBHC attitude 

One non-RCT (Ilic et al., 2013) and two RCTs (Ilic et al., 2015; Kulier et al., 2009) reported on 
EBHC attitude, but only Ilic et al. (2015) reported sufficient data to be included in a meta-analysis. 
EBHC attitude scores were significantly higher in the blended learning group compared to the face- 
to-face learning group (SMD 1.07 95%CI 0.57 to 1.58; 1 study, n=82) (Analysis 4 . 4 - 1 . The SMD of 
1.08 can be interpreted as a large effect size. 
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In Ilic et al. (2013) attitude scores in the blended learning group were significantly higher 
compared to the face-to-face group for the statement “I believe that I will use my EBM skills 
during my clinical career” (p=o.03), but not for the statement “I believe that practicing evidence 
based medicine is critical in being a good clinician” (p=o.o8). 

There was no statistically significant difference in attitudinal gain between blended learning and 
face-to-face learning groups in Kulier et al. (2009). 

4.3. 4.1. 5 EBHC behaviour 

One study (Ilic et al., 2015) reported on EBHC behaviour using the self-reported Evidence-based 
Practice Questionnaire, which contains six questions related to EBHC behaviour. Data was 
analysed for 17% (44/263) of participants from the blended learning group and 16% (38/234) of 
participants from the face-to-face learning group. 

EBHC behaviour scores were significantly greater for participants in the blended learning group 
(SMD 2.34 95%CI 1.72 to 2.96; 1 study, n=82) (Analysis 4.5) . The SMD of 2.35 can be interpreted 
as a huge effect. 

4.3.4.2 Secondary outcomes 

4.3.4.2.1 Satisfaction of students with method of learning 

One non-RCT (Ilic et al., 2013) and one RCT (Ilic et al., 2015) reported on satisfaction of 
participants with the method of learning. 

Ilic et al. (2013) found that learners in the blended learning group were significantly more satisfied 
with the method of learning compared to the face-to-face learning group, with three of the five 
statements related to satisfaction being statistically significant. Learners preferred the blended 
learning approach. 

Ilic et al. (2015) found that students preferred a blended learning approach. Learners felt that 
didactic lectures were useful for acquiring new knowledge, but that it did not matter whether these 
were delivered face-to-face or online. They also perceived small group activities to be useful in 
acquiring new skills, which also motivated them to apply these in the clinical context. Learners in 
the face-to-face learning group, however, found the content to be “dense and dry” and felt that the 
learning approach only supported superficial learning. 

4.3. 4.2.2 Satisfaction of educator with method of learning 
No studies reported on this outcome. 

4.3.4.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3.4.2.4 Barriers to the method of learning EBHC 
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One RCT reported on the enablers of the method of EBHC learning (Ilic et al., 2015). These 
included variations in methods of implementation of learning across study sites and the difficulty 
to apply the learned concepts in clinical practice. Learners also felt that EBHC teaching would have 
greater value in subsequent clinical years. 

4-3-4-Z-5 Cost 

No studies reported on this outcome. 

4.3.4.2.6 Attrition of learners 

One non-RCT (Ilic et al., 2013) and four RCTs (Brettle & Raynor, 2013; Ilic et al., 2015; Kulier et 
al., 2009; Saunders et al., 2016) reported on attrition of learners. In Ilic et al. (2013) there was no 
loss to follow-up in either group. The four RCTs were included in the random-effects meta-analysis 
(Analysis 4.61 and showed similar attrition rates between blended learning and face-to-face 
learning groups (RR 1.50 95%CI 0.79 to 2.85; n=729; Tau 2 = 0.17; Chi 2 = 4.59, P=0.20; I 2 = 35%). 

4.3.4.2.7 Learner adherence 

No studies reported on this outcome. 

4.3. 4.2.8 Evidence-based practice 
No studies reported on this outcome. 

4.3. 4.2 . 9 Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.4.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.4.2.11 Individual health outcomes 
No studies reported on this outcome. 

4.3.4.2.12 Population health outcomes 
No studies reported on this outcome. 

4.3.5 Blended learning vs pure e-learning 

One non-RCT (Fernandez et al., 2014) and two RCTs (Kamin et al., 2001; Kulier et al., 2012) 
compared pure e-learning to blended learning (Table 12. 5) . In Fernandez et al. (2014), the pure e- 
learning intervention (DVD group) was compared to three control groups: the standard distance 
group (pure e-learning), the computer lab group (blended learning) and the didactic lectures group 
(blended learning). We included the two blended learning groups for this comparison. Overall, 
results favoured blended learning. 
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4'3'5‘1 Primary outcomes 


4.3. 5. 1.1 EBHC knowledge 

Two RCTs (Kamin et al., 2001; Kulier et al., 2012) reported on EBHC knowledge and were included 
in the random-effects meta-analysis (Analysis 5.1) . EBHC knowledge scores were significantly 
higher in the blended learning group compared to the pure e-learning group (SMD 0.69 95%CI 
0.40 to 0.99; 2 studies, 11=193). Results were consistent across the two studies with low levels of 
heterogeneity (Tau 2 = 0.00; Chi 2 = 0.97, P = 0.32; I 2 = 0%). The SMD of 0.69 can be interpreted 
as a medium effect size. 

4.3. 5. 1.2 EBHC knowledge and skills 
No studies reported on this outcome. 

4 - 3 - 5 - 1-3 EBHC skills 

One non-RCT (Fernandez et al., 2014) and one RCT (Kulier et al., 2012) reported on EBHC skills. 

In Fernandez et al. (2014), participants had to submit two assignments. The first assignment 
measured EBHC skills related to formulating PICO questions, searching the literature and 
identifying the level of evidence, while the second assignment measured critical appraisal skills. We 
included critical appraisal scores in the random-effects meta-analysis (Analysis .4.2) . Overall, there 
was no difference between blended learning and pure e-learning (SMD -0.53 95%CI -2.31 tol.25; 2 
studies, 11=218). There was a significant difference between subgroups (p<o. 00001, 1 2 =96.2%), 
with results for the non-RCT (Fernandez et al., 2014) favoring pure e-learning (SMD -1.46 95%CI - 
2.08 to -0.84) and results for the RCT (Kulier et al., 2012) favouring blended learning (SMD 0.36 
95%CI 0.05 to 0.67). 

We report the remaining results from Fernandez et al. (2014) related to PICO, searching and levels 
of evidence in Analysis 5.3 . Overall, results favoured the pure e-learning group. 

4.3.5.1.4 EBHC attitude 

Although authors of one RCT (Kulier et al., 2012) stated that they measured EBHC attitude, they 
only reported baseline scores, and not those at follow-up. 

4-3-5 -1-5 EBHC behaviour 

No studies reported on this outcome. 

4.3.5.2 Secondary outcomes 

4.3.5.2.1 Satisfaction of students with method of learning 
No studies reported on this outcome. 

4.3.5. 2.2 Satisfaction of educator with method of learning 
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No studies reported on this outcome. 


4.3.5.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3. 5. 2. 4 Barriers to the method of learning EBHC 
No studies reported on this outcome. 

4-3-5- 2 -5 Cost 

No studies reported on this outcome. 

4.3. 5. 2. 6 Attrition of learners 

One RCT (Kulier et al., 2012) reported on attrition of learners. In the blended learning group, 
25/123 (20.3%) of learners were lost to follow-up, compared to 13/81 (16%) in the pure e-learning 
group (RR 1.27, 95%CI 0.69 to 2.33, 1 study, n=204) (Analysis 5.4) . 

4.3.5.2.7 Learner adherence 

No studies reported on this outcome. 

4.3.5. 2. 8 Evidence-based practice 
No studies reported on this outcome. 

4.3. 5. 2. g Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.5.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.5.2.11 Individual health outcomes 
No studies reported on this outcome. 

4.3.5.2.12 Population health outcomes 
No studies reported on this outcome. 

4.3.6 Pure e-learning vs pure e-learning 

One non-RCT (Fernandez et al., 2014) and two RCTs (Brouwers et al., 2011; Macrae et al., 2004) 
compared pure e-learning to another form of pure e-learning (Table 12 . 61 . Brouwers et al. (2011) 
compared three intervention groups. The first group received an online tutorial on AGREE II and 
was granted access to a PDF copy of the AGREE II tool, the second group received the online 
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tutorial plus a practical exercise and feedback, and the third group received the AGREE II manual. 
For Fernandez et al. (2014), we included the comparison between the DVD group and the standard 
distance group. In MacRae et al. (2004) the intervention group participated in an online journal 
club that included an asynchronous discussion list, while the control group only received the 
articles per email and had access to electronic journals. 

4. 3. 6.1 Primary outcomes 

4.3.6.1.1 EBHC knowledge 

4.3. 6. 1.2 No studies reported on this outcome. EBHC knowledge and skills 
No studies reported on this outcome. 

4.3.6. 1.3 EBHC skills 

One non-RCT (Fernandez et al., 2014) and two RCTs (Brouwers et al., 2011; Macrae et al., 2004) 
reported on EBHC skills. We included critical appraisal skills for one non-RCT (Fernandez et al., 
2014) and one RCT (Macrae et al., 2004) in the random-effects meta-analysis (Analysis 6.T) . 

Results favoured the intervention (SMD 1.30 95%CI 0.68 to 1.98, 2 studies, n=ii9) with moderate 
levels of heterogeneity (Tau 2 = 0.12; Chi 2 = 2.45, P = 0.12; I 2 = 59%). The test for subgroup 
differences was not significant (p=o.i2, 1 2 =59.2%). The SMD of 1.30 can be interpreted as a very 
large effect. Interventions comprised a DVD containing all the learning material compared to the 
standard distance learning programme in Ferndandez et al. (2014) and an online journal club with 
an asynchronous discussion list compared to receiving the journal arcitles via email in MacRae 
(2004). 

Brouwers et al. (2011) calculated the distance function, which is the difference between participants 
and experts in scores for each domain of the AGREE II tool. There was no statistically significant 
difference in distance function between any of the three pure e-learning intervention groups. 

EBHC skills related to PICO, searching the literature and levies of evidence are reported in Analysis 
6.2 . Overall, participants in the DVD group achieved higher scores compared to the standard 
distance learning group. 

4.3.6.1.4 EBHC attitude 

No studies reported on this outcome. 

4.3. 6. 1.5 EBHC behaviour 

4.3. 6. 2 No studies reported on this outcome. Secondary outcomes 

4.3.6.2.1 Satisfaction of students with method of learning 

One study (Brouwers et al., 2011) reported on satisfaction with the method of learning. There was 
no statistically significant difference in satisfaction scores between the three pure e-learning 
groups. 
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4-3-6-2.2 Satisfaction of educators with method of learning 


No studies reported on this outcome. 

4.3. 6.2.3 Enablers of the method of learning EBHC 
No studies reported on this outcome. 

4.3.6.2.4 Barriers to the method of learning EBHC 
No studies reported on this outcome. 

4-3 -6. 2. 5 Cost 

No studies reported on this outcome. 

4-3.6.2.6 Attrition of learners 

Two studies (Brouwers et al., 2011; Macrae et al., 2004) reported on attrition rates. Random-effects 
meta-analysis showed that attrition rates were similar between groups (RR 1.43 95%CI 0.89 to 
2.31; n=l70; Chi 2 = 0.89, P=o.64; I 2 =o%) (Analysis 6.3) . 

4.3.6.2. y Learner adherence 

No studies reported on this outcome. 

4.3.6.2.8 Evidence-based practice 
No studies reported on this outcome. 

4.3. 6. 2. g Evidence-based guideline implementation 
No studies reported on this outcome. 

4.3.6.2.10 Health care delivery 

No studies reported on this outcome. 

4.3.6.2.11 Individual health outcomes 
No studies reported on this outcome. 

4.3.6.2.12 Population health outcomes 
No studies reported on this outcome. 
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5 Discussion 


5.1 SUMMARY OF MAIN RESULTS 


Twenty-four studies with a total of 3806 participants comprising 13 individually randomised trials, 
seven cluster RCTs and four non-RCTs met the inclusion criteria for our systematic review. 

Participants included under- and postgraduate students, as well as practicing health care 
professionals of various health professions with medicine being the most common profession 
(14/24 studies). Studies were mainly conducted in high-income countries including the USA, 
Australia and various countries in Europe with only two trials conducted in LMICs - one in the 
Philippines and one in multiple countries: Argentina, Brazil, Democratic Republic of Congo, India, 
Philippines, South Africa and Thailand. 

E-learning interventions were heterogeneous. Although the content of the interventions generally 
covered the first four steps of EBHC (asking questions, accessing the literature, critically appraising 
the literature, and applying the results), we identified 17 different categories of intervention 
components. The interventions of five studies included only one component (single intervention), 
while the remaining interventions comprised various components in combination and were 
considered to be multi-faceted. Duration of interventions ranged from a single, one-hour session to 
a journal club addressing one article per month and running over eight months. Ten studies 
explicitly referred to educational theories or learning approaches underpinning the interventions 
but did not report results in light of these. 

Studies mostly reported on EBHC knowledge and skills, with some reporting on EBHC attitude and 
only five assessing EBHC behaviour. Outcomes were measured at different time points and with 
various tools, including validated questionnaires, self-perceived knowledge scores, hypothetical 
assignments and critical appraisal of studies. Outcomes were typically measured and compared 
after the intervention. Most studies reported outcomes at follow-up, with only a few adjusting 
results for baseline outcome measurement. Secondary outcomes were rarely reported. Four studies 
reported on satisfaction of learning, while one study explored barriers to EBHC learning. Rates of 
the attrition of learners were reported in 16 studies and we were able to pool these results using 
meta-analysis. 

Due to these variations in populations, interventions and measures of outcome assessments across 
included studies, heterogeneity was considered to be substantial. We therefore pooled results in a 
random-effects meta-analysis using the SMD for continuous outcomes. Due to the limited number 
of studies included per comparison, we were unable to explore remaining heterogeneity through 
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subgroup analysis. Heterogeneity due to the different interventions needs to be taken into 
consideration when interpreting results. In an effort to tease out the differences and similarities 
between interventions, we compiled a matrix showing the different intervention components for all 
the included studies (Table Q.7) . The EBHC content of each intervention is summarised in Table 

We summarised results for each comparison in Tables 12.1 to 12.6 . We found that pure e-learning, 
compared to no learning, improved EBHC knowledge and attitude. However, studies in this 
comparison had high risk of selection and attrition bias, sample sizes were small and results 
imprecise. For each outcome, only one study provided data for the meta-analysis. 

Blended learning compared to no learning improved EBHC knowledge, EBHC knowledge and skills 
one month post-intervention, and 3+ months post-intervention. We did not find significant 
differences between groups for EBHC attitude and EBHC behaviour. One study assessed EBHC 
behaviour 3 months after the intervention and found a significant difference in scores, favouring 
the blended learning group. There was high risk of selection and/or attrition bias across studies, 
inconsistency in results and imprecision for most outcomes. 

There was no difference in results for pure e-learning compared to face-to-face learning for EBHC 
knowledge, skills and attitude. Results for outcomes in this comparison were more precise 
compared to other comparisons, there was high risk of attrition bias and moderate heterogeneity 
for the outcome EBHC knowledge, caused by one outlier. 

There was no difference in EBHC knowledge and skills scores when comparing blended learning to 
face-to-face learning. We found a significant difference between groups for EBHC attitude and 
behaviour, although only one study with high risk of attrition bias was included in the analysis. In 
addition, authors did not adequately adjust for clustering. 

Blended learning compared to pure e-learning improved EBHC knowledge. There was unclear risk 
of selection and attrition bias and high risk of recruitment bias and loss of clusters. For EBHC 
skills, the two studies included in the analysis had inconsistent results. While results for the non- 
RCT favoured pure e-learning, results for the cluster RCT favoured blended learning. Confidence 
intervals were thus very wide and the results imprecise. 

When comparing various pure e-learning interventions, the pooled effect was imprecise and 
heterogeneity between studies was high. One study compared a DVD containing recorded 
PowerPoints and tutorials, as well as access to online learning material (intervention) to a standard 
online distance learning programme. The other compared an online journal club with an 
asynchronous discussion list (intervention) to receiving the articles via email and access to journal 
articles. Studies had high risk of selection, attrition and detection bias. 


5.2 OVERALL COMPLETENESS AND APPLICABILITY OF EVIDENCE 


We found substantial heterogeneity between interventions related both to intervention 
components and delivery of interventions, as has been shown in a previously conducted systematic 
review (Sinclair et al 2016). We were unable to perform planned subgroup analysis on the different 
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dimensions of e-learning, due to the limited number of included studies per meta-analysis. Studies 
that reported significant differences between groups typically included multi-faceted interventions 
(interventions comprising more than one intervention component), intervention components that 
required participants to interact with one another (e.g. asynchronous discussion lists), and 
integration of learning with clinical practice by including e.g. mobile bedside learning, exercises 
related to clinical cases, and access to clinical facilitators. These findings are in line with 
international literature and recommendations on effective teaching of EBHC (Khan & 
Coomarasamy, 2006; Young et al., 2014). In addition, our results suggest that blended learning is a 
more effective strategy than pure e-learning, since it includes multiple components by definition 
and follows a “best of both worlds” approach. 

These suggestions also resonate with other literature on e-learning. A systematic review on 
instructional design variations in internet-based learning, found that interactivity, practice 
exercises, repetition and feedback appeared to be associated with improved learning outcomes 
(Cook et al., 2010). A review by Booth and colleagues (2009) found that presentation and design, 
flexibility, peer communication, support and knowledge validation was effective e-learning 
techniques for enhancing the learning experience of students. 

Educational context was poorly reported in all included studies. As learning does not occur in a 
vacuum, it is essential to take into consideration the context within which learning takes place 
(Figure 2) . E-learning is a complex process where the learner, the technology and the context 
interact (Sandars & Lafferty, 2010). Factors such as learner motivation and the presence of role 
models in the clinical field can impact significantly on the acquisition of knowledge and skills, as 
well as learners’ attitude towards EBHC. Unfortunately, educational context was poorly reported in 
all included trials and we were therefore unable to take setting, learner, institution and socio- 
economic background into consideration in interpreting the results. 

Although some studies did show significant differences in EBHC knowledge and skills scores 
between groups, the post-intervention scores for intervention groups were generally low, especially 
when measured with either the Fresno test or the Berlin test. Both these tools have been validated 
to measure all four steps of EBHC (Shaneyfelt et al., 2006) and are commonly used. In Dizon et al. 
(2014), for example, EBHC knowledge and skills were measured with an adapted Fresno test and 
the mean post-intervention score in the intervention group was 64.3/156 (41%). In Ilic et al. (2015), 
the Berlin test was used and the mean post-intervention score in the intervention group was 8.2/15 
(57%). This raises questions about the actual effectiveness of the interventions in terms of 
knowledge gain and whether obtaining half (or less than half) of the maximum score is considered 
adequate when learning a new skill. In addition, knowledge and skills scores are the measurable, 
direct effects of the intervention. Increased knowledge and skills do not automatically translate to 
behaviour change and evidence-informed decision-making, which would be the ultimately desired 
outcome (Figure 2) . 

EBHC behaviour is a complex outcome and it is difficult to measure it objectively (Shaneyfelt et al., 
2006; Strauss et al., 2004; Tilson et al., 2011). Only five of the included studies reported on this 
outcome. In two of these (Kok et al., 2012; Forsetlund et al., 2003), EBHC was measured 
objectively, while the other three (Ilic et al., 2015; Ramos-Morcillo et al., 2015; Dizon et al., 2014) 
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used self-reported measures. Tilson et al. (2011), as part of the Classification Rubric for EBP 
Assessment Tools in Education (CREATE) Framework, propose that EBP behaviour should be 
assessed through activity monitoring. They highlight the lack of adequate tools to measure EBP 
behaviour and recommend that “valid, practicable methods are needed for monitoring learners’ 
EBP behaviours that can be used for both formative and summative purposes”. Indeed they also 
encourage researchers to use a common set of tools when assessing outcomes related to EBHC. 

Satisfaction with learning was only addressed in four studies. Learner satisfaction is linked to 
learner motivation and drive and is thus an important factor to consider when developing EBHC 
modules. Our results were not conclusive regarding preferences of participants. In two studies, 
focus group discussions were held with some of the participants and reported on the results as part 
of the article (Ilic et al., 2013; Ilic et al., 2015), while two other studies (Bradley et al., 2005; Welch, 
Van Lunen, & Hankemeier, 2014) conducted qualitative interviews alongside the respective 
quantitative investigation, but published on these results in separate articles. Bradley and 
colleagues (2005) explored the experiences of students and tutors regarding directed and self- 
directed learning, but included data from 40% of students that did not participate in the trial. 
Welch and colleagues explored the perceived effectiveness of the e-learning module, but only 
included participants from the experimental group in the interviews (Welch, Van Lunen, 
Hankemeier, et al., 2014). 

Attrition rates were high for most studies, even though they were similar between groups for all 
comparisons. Attrition of learners poses a significant threat to sustainability of e-learning 
programmes, since initial enthusiasm to learn decreases over time, especially in qualified 
professionals, who have limited free time to spend on e-learning. The benefits of e-learning in 
terms of learning at one’s own convenience, time and place can become a disadvantage, especially 
if it is not linked to assessment or a formal certificate. 

The studies included in this review were mostly undertaken in high income countries, with only 
two studies conducted in LMICs. One study was conducted in the Philippines, while the other was a 
multisite RCT conducted in various LMICs. Problems with internet connectivity, availability of PCs 
and access to databases are important challenges that will need to be addressed when 
implementing e-learning programmes in these countries. The studies conducted in LMICs did not 
assess these factors, on the contrary, in Kulier et al. (2012) adequate access to computers and 
databases was a part of the eligibility criteria for participation. 


5.3 QUALITY OF THE EVIDENCE 


We included 13 individually randomised controlled trials, seven cluster RCTs and four non-RCTs. 
Most comparisons only comprised one or two studies. We summarised results for each comparison 
in Tables 12.1 to 12.6 and commented on limitations related to the analyses. We made judgements 
about risk of bias according to the EPOC criteria and made additional judgements on risk of bias 
for cluster RCTs. Overall, the risk of bias in included studies was moderate. Only eight of the 24 
included trials were judged as having low risk of selection bias, having used adequate methods for 
both random sequence generation and allocation concealment. In addition, baseline characteristics 
were either not reported or showed significant differences between groups for seven of the included 
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studies. Fourteen studies were judged as having high risk of attrition bias, since loss to follow-up 
rates were very high. When making judgements about blinding, we only considered blinding of 
outcome assessors (detection bias), since blinding of participants and personnel (performance 
bias) is usually not possible for educational interventions. Three studies were judged as having 
high risk of detection bias, while ten studies did not report on blinding and were judged as having 
unclear risk of detection bias. We were unable to make a judgement regarding contamination for 
most individually randomised trials, since authors did not report how contamination was avoided. 
Of the cluster RCTs, two trials did not adjust results for clustering, limiting their compatibility with 
individually randomised trials. One study was judged as having high risk of recruitment bias, since 
participants were recruited after randomisation of clusters. 

These limitations in quantity and quality of the evidence need to be taken into consideration when 
interpreting results. 


5.4 LIMITATIONS AND POTENTIAL BIASES IN THE REVIEW 
PROCESS 


We attempted to minimize bias in the review process. We conducted a comprehensive search 
across multiple medical and educational databases and did not apply any limits with regards to 
language or publication status. Two authors independently screened search results, selected 
studies for inclusion, extracted data and made risk of bias assessments. We also contacted authors 
in an attempt to obtain missing data. We were unable to produce funnel plots to assess reporting 
bias, since we did not include more than 10 studies per outcome in the meta-analysis. 

The main limitation of our review is that we were unable to address our secondary objectives, 
because our included studies provided insufficient information on educational context and 
implementation strategies. In addition, we were unable to perform planned subgroup analysis on 
the effects of specific dimensions of e-learning. This was due to inadequate reporting of the 
included interventions and the small number of included trials per comparison. Phillips et al. 
(2016) recently published a guideline for reporting evidence-based practice educational 
interventions and teaching (GREET), in which they highlight the need to report details of the 
intervention. These include describing the educational theories or approaches, the learning 
objectives and EBHC content, learning materials and educational strategies, modes of delivery and 
educational environment. Indeed, we planned to extract data related to all these issues, but found 
that interventions were generally poorly described. 


5.5 AGREEMENTS AND DISAGREEMENTS WITH OTHER STUDIES OR 
REVIEWS 


EBHC e-learning is embedded in two domains, namely teaching EBHC and e-learning. 

In terms of teaching EBHC, the findings of our review echo the results of an overview of systematic 
reviews on the effectiveness of EBHC teaching and learning, which included 16 systematic reviews 
(Young et al., 2014). This review recommends that teaching and learning strategies should be 
multi-faceted and clinically integrated and that they should incorporate assessment. We found that 
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effective EBHC e-learning interventions were multi-faceted, contained interactive components and 
were integrated into clinical practice. Blended learning appeared to be more effective than pure e- 
learning. 

The systematic review by Ilic & Maloney (2014) that aimed to determine which types of educational 
methods were most effective in increasing EBHC knowledge and skills included four of the studies 
included in our review (Bradley et al., 2005; Davis et al., 2007; Davis et ah, 2008; Schilling et ah, 
2006) but compared a variety of educational methods (e.g. lectures, workshops, small group 
learning, problem-based learning, online learning, computer-assisted learning) and did not focus 
on e-learning compared to other types of learning. They found no difference in knowledge and 
skills between various educational methods. 

In terms of e-learning, our review is also in accordance with the results of other relevant studies. 
When comparing e-learning (pure e-learning or blended learning) to no learning, our results were 
similar to those of other systematic reviews, favoring pure e-learning for knowledge and skills (Al- 
Shorbaji et ah, 2015; Cook et ah, 2008; Liu et ah, 2016) and blended learning for EBHC behaviour 
(Sinclair et ah, 2016). 

When comparing e-learning (pure e-learning and blended learning) to other methods of learning 
(face-to-face learning or pure e-learning), our results were similar to those from other systematic 
reviews for pure e-learning compared to face-to-face learning, showing no difference in EBHC 
knowledge, skills, attitude (Al-Shorbaji et ah, 2015; Cook et ah, 2008; Liu et ah, 2016) and 
behaviour (Sinclair et ah, 2016). Our results favoured blended learning for EBHC attitude and 
behaviour when compared to face-to-face learning, and for EBHC knowledge when compared to 
pure e-learning. 
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Authors’ conclusions 


6 


6.1 IMPLICATIONS FOR PRACTICE AND POLICY 


Our findings suggest that e-learning of EBHC, whether pure or blended, compared to no learning, 
improves EBHC knowledge and skills. We did not find a difference in these outcomes when 
comparing e-learning to face-to-face learning, suggesting that both methods of learning can be 
beneficial. It appears that blended learning, which typically comprises multiple intervention 
components, could be more effective than other types of learning in improving EBHC knowledge, 
skills, attitude and behaviour. 

These findings need to be considered in light of the limited number of studies per outcome in each 
comparison, risk of bias across studies and heterogeneous interventions, as well as inconsistent 
and imprecise results. 

Importantly, e-learning as such is not a panacea and the principles that apply to EBHC teaching in 
general (e.g. interactive, clinically integrated teaching) should be considered when developing e- 
learning training initiatives. Other factors such as resources, feasibility and preference of learners 
need to be taken into consideration when planning EBHC learning activities. 


6.2 IMPLICATIONS FOR RESEARCH 


Future research on EBHC e-learning should focus on the effectiveness of various e-learning 
components and should aim to identify a combination of minimum components for effective e- 
learning. Interventions should be based on suitable learning theories, and explicitly reported. In 
addition, studies should evaluate the educational context as part of the intervention, explore 
learners’ experience with learning and include outcomes related to short- and long term EBHC 
behaviour and cost of the intervention. Comprehensive reporting of these aspects in primary 
studies will be critical, if an updated systematic review is to provide more in-depth insights. There 
is a need to conduct studies on e-learning in LMICs in order to address the challenges unique to 
these settings. 
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9 Tables 


9.1 SEARCH STRATEGY 


Medline via 

PubMed 

(((((("computer-assisted” OR "computer assisted" OR "internet-based" OR 
"internet based" OR internet OR "web 2.0" OR "web-based" OR "web based" 

OR online OR electronic OR e-learning OR computer OR "computer-based" OR 
virtual OR "social media" OR mobile))) AND ((Education, Medical [MeSH] OR 
"medical education" OR Education, Medical, Graduate [MeSH] OR "graduate 
medical education" OR Education, Medical, Undergraduate [MeSH] OR 
"undergraduate medical education" OR Education, Medical, Continuing 
[MeSH] OR "continuing medical education" OR "continuing professional 
development" OR course OR training OR module OR workshop OR curriculum 
OR education OR instruction))) AND (((("critical appraisal" OR "journal 
club"))) OR ((((("evidence-based practice" OR "evidence based practice" OR 
"evidence-based medicine" OR "evidence based medicine" OR "evidence-based 
health care" OR "evidence-based healthcare" OR "evidence based health care" 

OR "evidence based healthcare" OR "evidence-based dentistry" OR "evidence 
based dentistry" OR "evidence-based nursing" OR "evidence based nursing")))) 
OR (((((("Evidence-Based Practice" [Mesh]) OR "Evidence-Based 
Practice/education"[Mesh]) OR "Evidence-Based Medicine"[Mesh]) OR 
"Evidence-Based Dentistry" [Mesh]) OR "Evidence-Based Nursing"[Mesh]))))) 

EMBASE via 

Ovid 

1. ^evidence based dentistry/ or ^evidence based medicine/ or ^evidence 
based practice/ or ^evidence based nursing/ 

2. *nursing education/ or ^paramedical education/ 

3. ^online system/ 

4. exp *medical education/ 

5. (evidence adj2 (health or medic* or nurs*OR pract* or dentist*)). ti,ab. 

6. (critical* adji apprais*).ti,ab. 

7. (journal adji club*).ti,ab. 

8. (continuing adji (profession* or medic* or nurs* or health*) adji 
(development or education)).ti,ab. 

9. ((medic* or nurs* or dentist* or health*) adj2 (educat* or train* or 
teach*)).ti,ab. 

10. 1 or 5 or 6 or 7 

11. 2 or 4 or 8 or 9 

12. "e-learning".ti,ab. 

13. ((online or web or internet or computer or virtual or media or mobile or 
electronic*) adj2 (learn* or teach* or educat* or course or workshop or 
train* or instruct* or curricul* or module*)).ti,ab. 

14. 3 or 12 or 13 

15. 10 and 11 and 14 

16. ("computer-assisted" or "computer assisted" or "internet-based" or 
"internet based" or internet or "web 2.0" or "web-based" or "web based" or 
online or electronic or e-learning or computer or "computer-based" or 
virtual or "social media" or mobile).ti,ab. 

17. ("medical education" or "graduate medical education" or "undergraduate 
medical education" or "continuing medical education" or "continuing 
professional development" or course or training or module or workshop or 
curriculum or education or instruction).ti,ab. 
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18. ("critical appraisal" or "journal club" or "evidence-based practice" or 
"evidence based practice" or "evidence-based medicine" or "evidence based 
medicine" or "evidence-based health care" or "evidence-based healthcare" 
or "evidence based health care" or "evidence based healthcare" or 
"evidence-based dentistry" or "evidence based dentistry" or "evidence- 
based nursing" or "evidence based nursing").ti,ab. 

19. 10 or 18 

20. 11 or 17 

21. 14 or 16 

22. 19 and 20 and 21 

CENTRAL, 

SCOPUS, 

CINAHL, Web 
of Science, 
PsychNet, 
ProQuest 

("computer-assisted" OR "computer assisted" OR "internet-based" OR 
"internet based" OR internet OR "web 2.0" OR "web-based" OR "web based" 

OR online OR electronic OR e-learning OR computer OR "computer-based" OR 
virtual OR "social media" OR mobile) AND ("medical education" OR "graduate 
medical education" OR "undergraduate medical education" OR "continuing 
medical education" OR "continuing professional development" OR course OR 
training OR module OR workshop OR curriculum OR education OR 
instruction) AND ("critical appraisal" OR "journal club" OR "evidence-based 
practice" OR "evidence based practice" OR "evidence-based medicine" OR 
"evidence based medicine" OR "evidence-based health care" OR "evidence- 
based healthcare" OR "evidence based health care" OR "evidence based 
healthcare" OR "evidence-based dentistry" OR "evidence based dentistry" OR 
"evidence-based nursing" OR "evidence based nursing" ) 


Return to text 
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9.2 RISK OF BIAS ASSESSMENT FOR RCTS AND NON-RCTS 


We answered the following questions with “yes” (low risk of bias), “no” (high risk of bias) or “unclear” (unclear risk of bias) to make 
judgments of risk bias for RCTs and non-RCTs. 


Domain 

Yes 

No 

Unclear 

Was the allocation 
sequence adequately 
generated? 

Low risk of selection bias if the sequence 
generation was truly random (e.g. 
computer-generated table of random 
numbers, tossing a coin) 

High risk of bias if sequence generation 
contained a non-random component (e.g. 
alternate randomisation, randomisation by 
birth date) 

Unclear risk of bias if the 
randomisation process was not 
clearly described 

Was the allocation 
adequately concealed? 

Low risk of selection bias if allocation was 
truly concealed (e.g. central allocation of 
participants, use of sequentially numbered, 
opaque, sealed envelopes) 

High risk of bias if the allocation process 
was not concealed (e.g. open 
randomisation, unsealed or non-opaque 
envelopes). We will score CBAs as “high 
risk” 

Unclear risk of bias if the process 
of concealing allocation was not 
described sufficiently to make a 
judgement 

Were baseline outcome 
measurements similar? 

Low risk of bias if performance or patient 
outcomes were measured prior to the 
intervention, and no important differences 
were present across study groups. We will 
score RCTs as having low risk of bias if an 
imbalance is present, but authors 
appropriately adjusted the results (e.g. 
Analysis of covariance). 

High risk of bias if important differences 
were present and not adjusted for in 
analysis. 

If RCTs have no baseline measure 
of the outcome, we will score the 
study as having unclear risk of 
bias. 

Were baseline 
characteristics similar? 

Low risk of bias if baseline characteristics of 
the study and control providers are 
reported in the study and are similar. 

High risk of bias if there is no report of 
participant characteristics in the text or 
tables; or if there are differences between 
the control and intervention providers. 

Unclear risk of bias if 
characteristics were not clearly 
reported in the paper (e.g. 
characteristics are mentioned in 
the text but no data were 
presented). 

Were incomplete outcome 
data adequately addressed? 

Low risk of attrition bias if there was no 
missing data or if missing data was 
balanced across groups 

High risk of bias if there was missing data 
or if missing data was more prevalent in 
one of the groups and likely to bias the 
results 

Unclear risk of bias if it is not 
specified in the paper. We will not 
assume a 100% follow-up rate, 
unless it is explicitly stated. 

Was knowledge of the 
allocated interventions 
adequately prevented 
during the study? 

Low risk of detection bias if they were blind 
to knowledge about which intervention the 
participants received; or if outcomes were 
objective 

High risk of bias if blinding was absent 

Unclear risk if blinding was not 
specified in the paper 


74 


The Campbell Collaboration | www.campbellcollaboration.org 


Was the study adequately 
protected against 
contamination? 

Low risk of bias if allocation was by 
community, institution or practice and it is 
unlikely that the control group received the 
intervention 

High risk of bias if it is likely that the 
control group received the intervention 

Unclear risk of bias if 
professionals were allocated 
within an institution or practice 
and it is possible that 
communication between 
intervention and control 
professionals could have occurred 
(e.g. physicians within practices 
were allocated to intervention or 
control) 

Was the study free from 
selective outcome 
reporting? 

Low risk of reporting bias if it is evident 
that all pre-specified outcomes have been 
reported on (e.g. all relevant outcomes in 
the methods section are reported in the 
results section). 

High risk of bias if it is evident that some 
outcomes were omitted in the report; 

Unclear risk of bias if it is unclear 
whether all outcomes have been 
reported on 

Was the study free from 
other risks of bias? 

Low risk of bias if there is no evidence of 
other risks of bias 

High risk of bias if there is evidence of other 
risks of bias (e.g. conflict of interest 

Unclear risk of bias if it is not clear 
from the paper whether other 
biases are present 


Return to text 
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9-3 CHARACTERISTICS OF INCLUDED STUDIES 


l. Bergold 2013 


Methods 

RCT - wait list control group 

Participants 

• Medical junior doctors, continuing medication (interns) 

• n=i20 

• Inclusion criteria: 

Medical doctors at start of their work at a university hospital during 
first year (Johann Wolfgang Goethe University, Frankfurt) 

Interventions 

Intervention: E-learning (blended) 

• German-language self-directed, independent and tutor-assisted online 
learning course comprising five sequential modules and designed within EU 
EBM project 

• Multifaceted intervention, integrated EBHC learning 

• Individual learning, blended learning 

• asynchronous 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

• Course delivered over a period of 5 weeks (self-directed) 

• Learning theory: 

SPICES framework 

Participant-centred approach 

Distributed learning 

Control: No learning 

• no learning (wait list) 

Outcomes 

1. EBHC knowledge 

Knowledge score (0-13), assessed through 13 module-specific 
questions in sets 1 and 2. Validated questions adapted and pilot- 
tested; questions sets comparable 
measured pre- and post-course (at 3, 6, and 12 months) 

2. Satisfaction with learning 

Post course 

Notes 

• Country: Germany 

• Conflict of interest: not mentioned 

• Ethics approval: not mentioned 

• Funding: Johann Wolfgang Goethe University, Frankfurt 

• Author contacted via email to request SDs for mean values in Table 1, and 
knowledge scores at 0 months for the control group. No response 


1.1 Risk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Telephone-randomised wait-list design; computer- 
generated randomisation list 

Was the allocation adequately 
concealed? 

Yes 

Participants immediately randomised after 
recruitment; provision of recruitment information 
to external site (by telephone), not involved with 
study 

Were baseline outcome 
measurements similar? 

Yes 

Intervention group: median 5 (Cl 4-6); mean 4.9 
Control group: median 4 (Cl 4-5); mean 4.9 
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no statistical tests performed; no narrative 
description of baseline results 

Were baseline characteristics 
similar? 

Unclear 

No table of baseline characteristics. Information on 
age and gender of participants in text only 

Were incomplete outcome data 
adequately addressed? 

Yes 

Loss to follow-up was minimal and similar in both 
groups: 

Intervention: 3% 

Control: 6% 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Unclear 

Blinding of researchers, participants and outcome 
assessors not described at all; blinding of 
participants highly unlikely, given waitlist design 

Was the study adequately protected 
against contamination? 

No 

Relatively small participant group, all young newly 
recruited medical doctors who probably interact 
frequently with their peers 

Was the study free from selective 
outcome reporting? 

Yes 

All outcomes pre-specified in the methods section 
reported on 

Was the study free from other bias? 



Unclear 

Possible selection/response bias: 120 out of 219 
eligible young medical doctors agreed to 
participate and may have more time or be more 
passionate about learning than non-participants 


2. Bradley 2005 


Methods 

RCT 

Participants 

• Undergraduate medical students 

• 11=175 

• Inclusion criteria: 

All students that attended the first day of the 10 th semester or gave reason for 
absence in advance and completed consent and baseline characteristic forms at 
the University of Oslo 

Interventions 

Intervention: E-learning 

• EBHC course 

Self-directed: Computer-assisted (CD-ROM) on five steps of EBM 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, pure e-learning learning 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the process of EBHC 

• Execution: 

• Duration: 2 weeks 

• Intensity: 5 times 

• Dose: half a day 

• Timing: First two weeks of the 10th semester (20 weeks) 

• Learning theory: 

Encouraged to be autonomous in their learning and to choose the tie, place 
and speed of learning (distributed learning) 

• Control: face-to-face learning 

• Workshop on five steps of EBM 

directed learning 

• Singe intervention, stand-alone EBHC learning 

EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 
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Evaluating the process of EBHC 

• Execution: 

• Duration: 2 weeks 

• Intensity: 5 times 

• Dose: half a day 

• Timing: First two weeks of the 10th semester (20 weeks) 

• Learning theory: 

Social learning theory 

Outcomes 

1. EBHC knowledge 

MCQ with 6 stem questions and 3 sub-questions (validated questionnaire 
by Taylor et al.) 

post-test only at week 20 (end of semester) - compulsory examination 

2. EBHC skills 

Critical appraisal skills; students had to critically appraise a paper. 

Checklist given to students 

post-test only at week 20 (end of semester) - compulsory examination 

3. EBHC attitude 

Questionnaire consisting of 7 statements which student had to rate using a 
Likert scale (validated questionnaire by Taylor et al.) 

Completed during week 3-17, non-compulsory 

Notes 

• Country: Norway 

• Conflict of interest: yes. none declared 

• Ethics approval: approval obtained 

• Funding: no funding received 


2.1 Risk of bias table 


Bias 

Authors’ 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Authors used random number tables 

Was the allocation adequately 
concealed? 

Yes 

Numerically ordered, sealed, opaque envelopes 
were used 

Were baseline outcome 
measurements similar? 

Unclear 

No baseline measurement of outcomes 

Were baseline characteristics similar? 

Yes 

There were more female students in the directed 
group, but we do not think this would have an 
impact on the results. All other baseline 
characteristics were similar 

Were incomplete outcome data 
adequately addressed? 

Yes 

Loss to follow up was minimal and similar for both 
groups: 

For knowledge questionnaire: 

Intervention: 2% 

Control: 6% 

Skills questionnaire: 

Intervention: 4% 

Control: 2% 

Attitude questionnaire: (this was a secondary 
outcome) 

Intervention: 33% 

Control: 54% 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Yes 

Blinding of participants and teachers not possible. 
Outcome assessors were blinded 

Was the study adequately protected 
against contamination? 

Unclear 

Attendance registers were kept. 2 of the self- 
directed group attended 1 session of the directed 
group. Difficult to prevent directed group from 
accessing CD ROMs in self-directed group 
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Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other sources of bias identified 



3. Brettle 2013 


Methods 

RCT 

Participants 

• Nurses, undergraduate 

• n =77 

• Inclusion criteria: 

New intake in March 2008 (Students not contaminated by previous 
IL sessions, studying a module that required a large element of IL) 
at Salford University 

Interventions 

Intervention: E-learning 

• Online information literacy tutorial 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, blended learning 

• asynchronous 

• EBHC components: 

Accessing the literature 

Execution: 

• Duration: not described 

• Intensity: 2 sessions, 1 month apart. 

• Dose: 1 hour each for the initial session and follow-up session 

• Timing: First module of foundation training 

• Learning theory: 

not mentioned 

Control: face-to-face learning 

• Face-to-face session on information literacy 

• Singe intervention, stand-alone EBHC learning 

• EBHC components: 

Accessing the literature 

Execution: 

• Duration: not described 

• Intensity: 2 sessions, 1 month apart. 

• Dose: 1 hour each for the initial session and follow-up session 

• Timing: First module of foundation training 

• Learning theory: 

not mentioned 

Outcomes 

1. EBHC skills: 

Search skills (score out of 10) 

Pre- and post-test for initial and follow-up sessions 

Notes 

• Country: UK 

• Conflict of interest: not declared 

• Ethics approval: approval obtained 

• Funding: not mentioned 


3.1 Risk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Students were allocated using an online random 
number generator 

Was the allocation adequately 
concealed? 

Unclear 

Authors report that the allocation was concealed but do 
not describe the method used 
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Were baseline outcome 
measurements similar? 

Yes 

No significant difference between baseline scores 

Were baseline characteristics 
similar? 

Unclear 

No table of baseline characteristics provided, only 
description in text 

Were incomplete outcome data 
adequately addressed? 

Yes 

Loss to follow-up similar in both groups: 

Intervention: 12.5% 

Control: 9% 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Unclear 

Participants could not be blinded and it is not 
mentioned if the outcome assessors were blinded 

Was the study adequately protected 
against contamination? 

Unclear 

Control group participants were granted access to the 
online material and intervention group participants 
had face-to-face sessions. But participants were 
recruited from the same institutions. 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported 

Was the study free from other bias? 

Yes 

No other source of bias identified 


4 * Brouwers 2011 


Methods 

RCT 

3 groups 

Participants 

• Participants from various backgrounds (doctors, nurses, physiotherapists, 
occupational therapists, methodologists, policy makers and trainees) 

• n=87 

• Inclusion criteria: 

No or limited experience to the original AGREE instrument or the 
AGREE II. Participants were sought from guideline programs, 
professional directories, and the Guideline-International-Network 
(G-I-N) community 

Interventions 

Intervention: E-learning 

Group 1: 

• Online tutorial on AGREEII (multimedia tutorial presentation) and granted 
access to a PDF copy of the AGREEII to review 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, Pure e-learning 

• asynchronous 

• EBHC components: 

Critical appraisal 

• Execution: 

Once off (presentation was 7 minutes long) 

• Learning theory: 

Not mentioned 

Group 2: 

• Online tutorial on AGREE II tool plus exercise and immediate feedback as 
well as formative feedback if scores fell out of range 

• Multifaceted intervention, stand-alone EBHC learning 

• Individual learning, Pure e-learning 

• asynchronous 

• EBHC components: 

Critical appraisal 

• Execution: 

Once off: presentation was 7 minutes long, not reported how long it 
took participants to complete exercise. 

• Learning theory: 

Not mentioned 

Control: e-learning 

Group 3: 
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• Participants were given AGREE II manual 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, pure e-learning 

• EBHC components: 

Critical appraisal 

• Execution: 

Self-directed 

• Learning theory: 

Not mentioned 

Outcomes 

1. EBHC skills 

Performance (using AGREE II to critically appraise a PG) directly 
after intervention (post-test only) 

2 . Satisfaction with learning (directly after intervention) 

Notes 

1 11 

• Country: Canada 

• Conflict of interest: declared no conflicts of interest 

• Ethics approval: approval obtained 

• Funding: Canadian Institutes of Health Research 


4*1 Risk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Computer-generated randomisation sequence 

Was the allocation adequately 
concealed? 

Unclear 

Not mentioned 

Were baseline outcome 
measurements similar? 

Unclear 

Post-test only 

Were baseline characteristics 
similar? 

No 

Participants with PhDs: 

Group l: 25%; Group 2: 15%; Group 3: 5% (control) 

% with health research methods training: 

Group 1: 85%; group 2: 85%; group 3 (control): 

100% 

Use of AGREE as a tool to evaluate PG - Never: 
Group 1: 71%; Group 2: 61%; Group 3: 48% 

(control) 

Use of AGREE II as a tool to inform PG reporting - 
Never: 

Group 1: 97%; Group 2: 100%; Group 3: 84% 
(control) 

Were incomplete outcome data 
adequately addressed? 

No 

Loss to follow up: 

Group 1: 17% 

Group 2: 23% 

Group 3: 14% (control) 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Unclear 

Participants were blinded to study conditions, 
blinding of outcome assessors not reported 

Was the study adequately protected 
against contamination? 

Yes 

All three groups had online learning and 
participants did not interact with each other and 
came from various organisations and backgrounds 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other bias identified 
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5. Davis 2007 


Methods 

RCT 

Participants 

• Medical doctors (interns), continuing medical education 

• 11=55 

• Inclusion criteria: 

Newly qualified doctors (interns) in six postgraduate centres in the 
UK West Midlands participating in the foundation program for 
newly qualified doctors 

Interventions 

Intervention: E-learning 

• Short, computer-based session on EBM (CD-ROM) consisting of recorded 
PowerPoint presentations 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, pure e-learning 

• asynchronous 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

• Intensity: a single session 

• Dose: 60 minutes (40 minutes for the session and 10 min each for the pre- 
and post-questionnaire) 

• Timing: Foundation training of newly qualified doctors 

• Learning theory: 

Not mentioned 

Control: face-to-face learning 

• Face-to-face lecture on EBM with similar content, structure and duration as 
intervention, using same PowerPoint slides and the same tutor 

• Singe intervention, stand-alone EBHC learning 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

• Intensity: a single session 

• Dose: 60 minutes (40 minutes for the session and 10 min each for the pre- 
and post-questionnaire) 

• Timing: Foundation training of newly qualified doctors 

• Learning theory: 

Not mentioned 

Outcomes 

1. EBHC knowledge 

Questionnaire based on validated assessment tools (Berlin and 
Fresno) 5 questions with pre-determined marking scheme (two 
structured questions and 3 MCQ) 

Immediately pre- and post-intervention 

2. EBHC attitude 

Questions based on validated assessment tools, 6 questions on five 
point Likert scale 

Immediately pre- and post-intervention 

Notes 

• Country: UK 

• Conflict of interest: yes, no conflicts of interest declared 

• Ethics approval: ethics exemption obtained 

• Funding: West Midlands Deanery 
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5*1 Risk of bias table 


Bias 

Authors’ 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

|Yes 

Computer-generated sequence 

Was the allocation adequately 
concealed? 

Yes 

1 

Sealed envelopes coded by third party, but not 
mentioned whether they were opaque and sequentially 
numbered 

Were baseline outcome 
measurements similar? 

Yes 

Yes, Intervention group had an average score of 69%; 
comparison group had an average score of 63% (no p- 
values or confidence intervals reported) 

Were baseline characteristics 
similar? 

No 

The table of baseline characteristics does not contain 
results of a statistical test. More participants (almost 
double the amount) in the lecture group had education 
in epidemiology and statistics 

Were incomplete outcome data 
adequately addressed? 

Yes 

1 1 

All participants accounted for, no loss to follow up 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

|Yes 

Not possible to blind teachers and students, but 
outcome assessors were blinded 

Was the study adequately 
protected against contamination? 

Yes 

Both sessions were delivered at the same time, in the 
same institution. Pre and post-test questionnaires 
completed during this time 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 

bias? 

1 

Yes 

No other bias identified 


6. Davis 2008 


Methods 

RCT 

Participants 

• Medical doctors, undergraduate 

• n=229 

• Inclusion criteria: 

Not described, undergraduate medical students at the University of 
Birmingham Medical School 

Interventions 

Intervention: E-learning 

• Short, computer-based session on EBM (CD-ROM) consisting of recorded 
PowerPoint presentations 

• Singe intervention, stand-alone EBHC learning 

• Individual learning, pure e-learning 

• Asynchronous 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

• Intensity: a single session 

• Dose: 60 minutes (40 minutes for the session and 10 min each for the pre- 
and post-questionnaire) 

• Timing: Foundation training of newly qualified doctors 

• Learning theory: 

not mentioned 

Control: face-to-face learning 

• Face-to-face lecture on EBM with similar content, structure and duration as 
intervention, using same PowerPoint slides and the same tutor 
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• Singe intervention, stand-alone EBHC learning 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

• Intensity: a single session 

• Dose: 60 minutes (40 minutes for the session and 10 min each for the pre- 
and post-questionnaire) 

• Timing: Foundation training of newly qualified doctors 

• Learning theory: 

Not mentioned 

Outcomes 

1. EBHC knowledge 

Questionnaire based on validated assessment tools (Berlin and 
Fresno) 5 questions with pre-determined marking scheme (two 
structured questions and 3 MCQ) 

Immediately pre- and post-intervention 

2. EBHC attitude 

Questions based on validated assessment tools, 6 questions on five 
point Likert scale 

Immediately pre- and post-intervention 

Notes 

• Country: UK 

• Conflict of interest: not mentioned 

• Ethics approval: unclear - authors mentioned that study was approved by 
Birmingham medical school. Not clear whether this refers to ethics approval 

• Funding: not described 


6.1 Risk of bias table 


Bias 

Authors' 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Computer-generated sequence 

Was the allocation adequately 
concealed? 

Yes 

Sealed envelopes coded by third party, but not 
mentioned whether they were opaque and sequentially 
numbered 

Were baseline outcome 
measurements similar? 

Yes 

Yes, they seem similar: Intervention group: 61%; control 
group: 63% (no p-values or confidence intervals 
reported) 

Were baseline characteristics 
similar? 

Unclear 

The table of baseline characteristics does not contain 
results of a statistical test. More participants in the 
lecture group had education in epidemiology and 
research methods. Not sure whether this is statistically 
different 

Were incomplete outcome data 
adequately addressed? 

No 

In the intervention (computer-based) group, loss to 
follow-up was 39% compared to 5% in the control 
(lecture-based) group 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Yes 

Not possible to blind teachers and students, but outcome 
assessors were blinded 

Was the study adequately 
protected against 
contamination? 

Yes 

Both sessions were delivered at the same time, in the 
same institution. Pre and post-test questionnaires 
completed during this time 

Was the study free from 
selective outcome reporting? 

Yes 

All pre-specified outcomes reported on 
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Was the study free from other Yes 
bias? 


No other bias identified. 


7. Dizon 2014 


Methods 

RCT 

Participants 

• Physiotherapists, continuing medical education 

• 11=54 

• Inclusion criteria: 

Licensed Physiotherapists in the Philippines, no previous formal 
training in EBP 

Interventions 

Intervention: E-learning (blended) 

• EBP training (face-to-face) plus EBP checklist and online support (access to 
lectures and reference materials; seek assistance in searching for evidence to 
answer clinical questions; assistance in validity assessments through critical 
appraisal of the evidence found; ask any queries and provide feedback) 

• Multi-faceted intervention; integrated EBHC learning 

• Individual learning, blended learning 

• asynchronous 

• EBHC components: 

Epidemiology 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution (workshop): 

• Duration and intensity: one day consisting of 6 lectures interspersed with 4 
practical sessions 

Execution (online support): 

• Duration: 3 months 

• Intensity and dose: self-directed, as needed by participant. 

• Learning theory: 

Adult learning theory 

Control: no learning 

• Wait list 

Outcomes 

1. EBHC knowledge: 

Adapted Fresno test: 156 max score 

Pre- and post-intervention, and at 3 months post intervention 

2. EBHC skills: 

Adapted Fresno test 

Pre- and post-intervention, and at 3 months post intervention 

3. EBHC attitude 

Validated questionnaire on EBP attitudes, 3 point Likert scale 

Pre- and post-intervention, and at 3 months post intervention 

4. EBHC behaviour 

Measured by activity diaries - logging activities used to find an 
answer to a case they were faced with 

Measured 3 months post intervention 

Notes 

• Country: Philippines 

• Conflict of interest: yes, none declared 

• Ethics approval: approval obtained 

• Funding: Philippine council for health research and development - 
department of Science and Technology 

• Authors contacted to obtain mean scores, received response 
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7.1 Risk of bias table 


Bias 

1 1 

(Authors' 

[judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Computer generated random numbers 

Was the allocation adequately 
concealed? 

Yes 

Done by an independent researcher, that allocated 
randomisation sequence to list of participants. “Allocation 
Iwas concealed from the researchers of this study” 

Were baseline outcome 
measurements similar? 

Yes 

Yes, no significant differences between baseline knowledge 
scores 

Were baseline characteristics 
similar? 

Unclear 

Significant difference btw groups for years in practice and 
age 

Were incomplete outcome data 
adequately addressed? 

No 

3 month loss to follow-up in intervention group: 44%, 
control group: 59% 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Yes 

Outcome assessors blinded 

Was the study adequately 
protected against 
contamination? 

Unclear 

Participants recruited from a database of physical 
therapists, the network of the Philippine physical therapy 
association and a list of hospitals. Not clear whether 
participants had access to each other and discussed the 
study 

Was the study free from 
selective outcome reporting? 

Yes 

All pre-specified outcomes reported 

Was the study free from other 
bias? 

Yes 

No other bias identified 


8. Fernandez 2014 


Methods 

• Non-RCT 

• Setting: University of Sydney, University of Hong Kong 

• Learner context: 

Students in Hong Kong probably have less background knowledge 
in EBHC 

• Socio-economic context: high-income countries 

Participants 

• Nurses, postgraduate 

• n= 186 

• Inclusion criteria: 

All students undertaking the EBN unit in the second session of 2010 
and both sessions in 2011 

Interventions 

Intervention: (e-learning) 

Group 1 (n=28) 

• Evidence-based practice (EBP) DVD: demonstrations of EBP skills provided 
on a DVD - simulation of computer lab workshops. 

Standalone EBHC learning, multi-faceted intervention 

Pure e-learning, individual learning 

Asynchronous 

• EBHC components: 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the EBHC process 

• Execution: 

EBHC module 
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Outcomes 

Duration and dose: 15 weeks, 10 hours per week in the Spring and 
Autumn semesters 

• Learning theory: 

Constructivist model of learning: new information is absorbed by 
learners by building on pre-existing knowledge and learners accept 
responsibility for their own learning and are self-motivated to 
engage in deeper learning 

Control: 

Group 2 (n=36), e-learning 

• Standard distance method: access to e-learning site and learning package 

standalone EBHC learning, multi-faceted intervention 
pure e-learning, individual learning 

Execution: 

• EBHC module 

• Duration and dose: 15 weeks, 10 hours per week in the Spring and Autumn 
semesters 

Group 3 (n=24), blended learning: 

• Computer Lab teaching method (on campus) - practical interactive 
computer-based training with immediate feedback plus standard access to e- 
learning site and learning packages) 

Standalone EBHC learning, multi-faceted intervention 

Blended learning, individual learning 

Execution: 

• EBHC module 

• Duration and dose: 15 weeks, 10 hours per week in the Spring and Autumn 
semesters 

• Additional workshop : 

• Intensity and dose: 3 workshops in the 15 week period, @ 2 hours each 

Group 4 (n=99) blended learning 

• Face to face didactic classroom teaching method - classroom teaching with 
access to e-learning site and learning package 

• Standalone, multi-faceted learning 

• Blended learning, individual learning 

• Asynchronous 

Execution: 

• EBHC module 

• Duration: 15 weeks 

• Intensity and dose: Lectures in week 1,2, 6, 7, 3.25 hours each; tutorials in 
week 4 and 9, 2.5 hours each 

• Delivery agent: lecturer 

• EBHC components (all control groups) 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the EBHC process 

1. EBHC knowledge and skills 

Assessed using students’ assignment marks 
post-test only 

Notes 

• Country: Australia and Hong Kong 

• Conflict of interest: not reported 

• Ethics approval: approval obtained 

• Funding: College of Health and Science University of Western Sydney 

• Authors contacted: No 
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8.1 Risk of bias table 


Bias 

Authors’ 

[judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

No 

Non-randomised study 

Was the allocation 
adequately concealed? 

No 

Non-randomised study 

Were baseline outcome 
measurements similar? 

Unclear 

No baseline measurement of outcomes 

Were baseline characteristics 
similar? 

Unclear 

No table or description of baseline characteristics, but one arm 
of the study (face-to-face didactic lecture group) was based in 
Hong Kong, where learners are not necessarily English mother 
tongue speakers and where previous teaching in EBP is 
unlikely. Difficult to compare to Australian setting 

Were incomplete outcome 
data adequately addressed? 

Unclear 

No flow diagram or description of participant flow or follow up 

Was knowledge of the 
allocated interventions 
adequately prevented during 
the study? 

No 

Some of the outcomes were assessed by study authors 

1 1 

Was the study adequately 
protected against 
contamination? 

Unclear 

No measures described to prevent contamination 

Was the study free from 
selective outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from 
other bias? 

Unclear 

Sample sizes: group that had face-to-face teaching three times 
as big as other groups. When comparing DVD group to this 
group, there might be issues with statistical analysis 


9. Forsetlund 2003 


Methods 

• RCT 

• Duration of study: April 1999 to January 2001 

• Setting: workplace (not specified) 

• Learner context: 

Background knowledge of EBHC: In the intervention group, 24% 
had attended session(s) on searching and 42% on critical appraisal. 

In the control group, 23% had attended session(s) on searching and 
30% on critical appraisal 

• Socio-economic context: 

17/8 (discrepancy btw table and text) physicians (7 in intervention 
group and 10 in control group) did not have access to internet and 
were sent copies of the reports that were made by the team in the 
web-based question and answer service 

Participants 

• Medical doctors (Public Health), continuing medical education 

• n= 148 

• Inclusion criteria: 

All public health physicians working in municipalities in Norway 
that had more than 3000 inhabitants 

Interventions 

Intervention (n=73), blended learning 

• EBM workshop with access to databases, library service and participation in 
asynchronous discussion list 

• Multi-faceted intervention, integrated learning 

• Blended learning, collaborative learning 
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• Asynchronous discussion list 

• EBHC components: 

- General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• Intervention lasted 1.5 years. Workshop was between 1 and 5 days long 

• Learning theory: Innovation-diffusion process 

• Delivery agent: 2 public health physicians and 2 librarians 

Control (n=75): no learning 

Outcomes 

Primary outcome: 

1. EBHC behaviour 

Measured by analyzing the contents of local health service reports 
and a hypothetical assignment, by a postal survey, a telephone 
survey and questionnaire 
post-intervention 

Secondary outcomes: 

1. EBHC knowledge 

Measured with questionnaire: Mean additive score of o=unknown, 
i=known, but not used, 2=read, 3=used in a public health decision- 
making situation 
post intervention 

2. EBHC attitude 

Measured with questionnaire: 7 point Likert scale 

Post intervention 

Notes 

• Country: Norway 

• Conflict of interest: reported, none declared 

• Ethics approval: not reported 

• Funding: The Norwegian Research Council 

• Authors contacted: No 


9.1 Risk of bias table 


Bias 

Authors* 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

1 

Done by an independent researcher using computer software 

Was the allocation adequately 
concealed? 

Unclear 

Not described 

Were baseline outcome 
measurements similar? 

[Unclear 

Not clear whether outcomes were measured at baseline 

Were baseline characteristics 
similar? 

No 

Imbalance for the following variables: sex, number of years as 
a public health physician, specialist status, previous exposure 
to courses in critical appraisal (Intervention group: 42%; 
control group: 30%) and number of advisory reports written 
during the previous year 

Were incomplete outcome 
data adequately addressed? 

No 

Loss to follow-up more than 20% for all outcome measures 

Was knowledge of the 
allocated interventions 
adequately prevented during 
the study? 

Yes 

Blinding of outcome assessors done 

Was the study adequately 
protected against 
contamination? 



Unclear 

Public health physicians from all Norwegian municipalities 
were invited to participate and individual randomisation was 
done. Contamination could have been avoided by using a 
cluster randomised design 
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Was the study free from 
selective outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other bias identified 


io. Hadley 2010 


Methods 

• Cluster RCT 

• Unit of randomization: Teaching hospitals 

• Duration of study: Between May and September 2007 

• Setting: 7 teaching hospitals in the UK West Midlands region 

• Socio-economic context: High-income country 

Participants 

• Medical doctors, continuing medical education 

• n= 237; 7 clusters 

• Inclusion criteria: Foundation year 2 doctors 

Interventions 

Intervention (n=i22; 4 clusters) e-learning 

• Clinically integrated teaching of EBM: Recorded PowerPoint slides on three 
modules on EBM (asking questions, accessing literature, appraising 
literature). Unlimited access for six weeks 

integrated learning, multifaceted intervention 
pure e-learning, individual learning 
asynchronous delivery 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

'Execution: 

• EBM course of 6 weeks duration; access according to individual needs 

• Learning theory: not mentioned 

• Delivery agent: self-directed 

Control (n=H5; 3 clusters), face-to-face learning: 

• Standalone, three hour face-to-face lecture (same content as intervention), 
same PowerPoint slides 

standalone learning, single intervention 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Execution: 

• Once off session of 3 hours 

• Learning theory: 

• Delivery agent: Lecturer 

Outcomes 

1. EBHC knowledge 

Measured with previously validated MCQs 

Pre- and post-intervention 

Notes 

• Country: UK 

• Conflict of interest: none declared 

• Ethics approval: not obtained ( not applicable according to authors) 

• Funding: European Union Leonardo da Vinci project 

• Authors contacted: no 


lO.iRisk of bias table 


Bias 

1 1 

lAuthors* 

[judgement 

Support for judgement 

Was the allocation sequence adequately 
generated? 

Yes - 

computer generated 
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Was the allocation adequately concealed? Unclear 

not described 

Were baseline outcome measurements 
similar? 

Unclear 

Baseline scores presented, but no significance 
test done 

Intervention: Mean 22.9 SD 7.0 n=88 

Control: Mean 24.7 SD 3.9 n=72 

Were baseline characteristics similar? 

[Unclear 

No table of baseline characteristics 

Were incomplete outcome data adequately 
addressed? 

No 

Loss to follow-up: 

Intervention: 25% 

Control: 31% 

Was knowledge of the allocated 
interventions adequately prevented during 
the study? 

Unclear 

Not able to blind teachers and participants. Not 
described whether outcome assessors were 
blinded 

Was the study adequately protected 
against contamination? 

Yes 

Cluster randomised study 

Was the study free from selective outcome 
reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other sources of bias identified 

Recruitment bias 

Low risk 

Randomisation was performed after recruitment 
and after consent was obtained 

Baseline imbalance 

Unclear 

No table of baseline characteristics 

Loss of clusters 

[Unclear 

not mentioned 

Incorrect analysis 

Low risk 

adjusted for clustering 

Compatibility with RCTs randomised by 
individuals 

Low risk 

yes, results are similar to RCTs with individual 
randomisation 


11. Horiuchi 2009 


Methods 

• RCT 

• Duration: August 2005 to November 2006 

• Setting: Nursing College in Tokyo (for face-to-face group) 

• Learner context: 

Background knowledge of EBHC: 

Face-to-face group scored higher on pre-test (73.5%) compared to 
web-based group (64.1%) p=o.i 

• Socioeconomic context: high-income country 

Participants 

• Nurses, continuing medical education 

• 11=93 

• Inclusion criteria: 

Registered nurses or midwives, with at least one year of clinical 
experience and presently working in a clinical area; understood and 
agreed with the aims of the study; had an interest in evidence-based 
nursing (EBN) techniques, and expressed a desire for ongoing 
education; access to a broadband internet connection either at 
home or work 

Interventions 

Intervention (n=45) e-learning: 

• E-learning of EBM divided into four parts, distributed according to individual 
progress 

Standalone learning, single intervention 

Individual learning, pure e-learning 

Asynchronous delivery 

• EBHC components: 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 
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1 

Outcomes 

Execution: 

• EBHC course, l month according to individual’s rate of progress. 

• Learning theory: 

Distributed learning: not explicitly mentioned but described 
through: participants were expected to access material when 
convenient for them 

• Delivery agent: self-directed 

Control (n= 48 ), face-to-face learning: 

• Face-to-face EBM teaching divided into one evening lecture per week 

Standalone learning, single intervention 

• EBHC components: 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• EBHC course of 4 weeks, 90 minute lecture once a week. 

• Learning theory: not mentioned 

• Delivery agent: Tutor (not described in detail) 

1. EBHC knowledge 

Measured through MCQ tests developed by authors, six questions, 
based on content covered 

Pre-test: after recruitment, before randomization; question and 
answer sheet sent via email, participants had to return answer sheet 
within one week. 

Post-test: sent via email after one month for intervention group, 
administered after 4 th lecture for face-to-face group 

2. Satisfaction of students with method of learning 

Measured access to and utilization of course 

Notes 

• Country: Japan 

• Conflict of interest: not reported 

• Ethics approval: approval obtained 

• Funding: Japanese ministry of education 

• Authors contacted: no 


n.iRisk of bias table 


Bias 

1 

Authors 1 

judgement 

Support for judgement 



Was the allocation sequence 
adequately generated? 

Yes 

Computerised random number generators 

Was the allocation adequately 
concealed? 

Yes 

"Concealed opaque envelopes containing the 
randomised allocations” produced by research assistant 
(not mentioned that they were sealed and sequentially 
numbered) 

Were baseline outcome 
measurements similar? 

Unclear 

Baseline scores: 

Intervention: Mean 64.1 SD 26.6 

Control: Mean 73.5 SD 20.7 
p=o.i 

Were baseline characteristics 
similar? 

Unclear 

Table of baseline characteristics does not contain p- 
values. Characteristics for “age” are not similar, the web- 
based group seems to be older 

Were incomplete outcome data 
adequately addressed? 

No 

31% lost to follow up in face-to-face group; 18% lost to 
follow-up in e-learning group 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Unclear 

Blinding of participants and teachers not possible, 
blinding of outcome assessors not reported 
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■ ■ 

Was the study adequately 
protected against contamination? 

Unclear 

not described 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes were reported on 

Was the study free from other 
bias? 

Yes 

No sources of other bias identified 


12. Ilic 2013 


Methods 

• Non-RCT 

• Students attending the Peninsula clinical site received the intervention, 
students across remaining three sites received control 

• Duration of study: not reported 

• Setting: Monash University in Australia 

Participants 

• Medical doctors, undergraduate (2 nd year graduate MBBS programme) 

• n= 6 i 

• Inclusion criteria: 

Second year Monash graduate MBBS student at the time of the 
study 

• Exclusion criteria: 

Students who were unwilling to participate in the study or did not 
wish to provide consent 

Interventions 

Intervention (n=34 ): blended learning of EBM consisting of: 

• Workshop 

Block day 

• Online learning 

Access to learning content via the Monash University website 
Self-directed learning 

• Clinical activities 

Applying principles of EBM to patient case 

• Tutorials 

Presentation of patient case and EBM content 

Quasi-journal club 

• Integrated learning, multifaceted intervention 

• Collaborative learning, blended learning 

• Asynchronous delivery 

• EBHC components 

Biostatistics 

General searching skills 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

Course for 2 nd year medical students doing first clinical rotation 
(graduate students of MBBS programme) 

Ten 2 hour sessions 

Learning theory: 

Peer-to-peer learning 

Problem-based learning 

Control (n= 27 ): Face-to-face learning (didactic learning) 

• Classroom activities 

Ten 2 hour tutorials 

Presentations of EBM concepts (didactic lectures) 

Small group tasks 

Large group discussions 

Structured learning activities 

• Standalone learning, multifaceted intervention 
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• EBHC components 

Biostatistics 

General searching skill 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

• Course for 2nd year medical students doing first clinical rotation (graduate 
students of MBBS programme) 

• Ten 2 hour sessions 

Outcomes 

1. EBHC knowledge and skills 

Berlin questionnaire 

Two assessment tasks 

Post intervention only 

2. EBHC attitude 

Survey 

Post intervention 

Notes 

• Country: Australia 

• Conflict of interest: Declared: DI is the coordinator of the EBM program for 
the MBBS degree at Monash University. PF and EV coordinate the EBM 
teaching program delivered through the Gippsland Medical School. 

• Ethics approval: approval obtained 

• Funding: not reported 

• Authors contacted: no 


12.1 Risk of bias 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

No 

Non-randomised study. Students attending the 

Peninsula clinical site received the intervention, 
students across remaining sites received control 

Was the allocation adequately 
concealed? 

No 

Non-randomised study 

Were baseline outcome 
measurements similar? 

Unclear 

Outcomes were not measured at baseline 

Were baseline characteristics 
similar? 

No 

Not reported - no table of baseline characteristics. 

Were incomplete outcome data 
adequately addressed? | 

Yes 

No participants lost to follow-up 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

No 

“Both assignments were graded by EBM tutors 
participating in this study”. Not mentioned that they 
were unaware of group allocation 

Was the study adequately 
protected against contamination? 

Unclear 

Although students were at different sites, they were all 
part of the same programme and not mentioned how 
contamination was avoided 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other potential biases identified 


13. Ilic 2015 


Methods 




Cluster RCT 
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• Unit of randomisation: Tutorial group 

• Duration of study: not reported 

• Setting: Monash University in Australia and Malaysia 

Participants 

• Medical doctors, undergraduate 

• n=497 (24 clusters) 

• Inclusion criteria: 

Participants were third year medical students, who were all entering 
their first year of clinically based training and first year of formal 
EBM training (graduate entry or undergraduate) 

Interventions 

Intervention (n =263 ): blended learning of EBM consisting of: 

• Classroom activities 

10 2-hour teaching sessions: formal EBM concepts delivered by 
tutor/lecturer. Sessions commence with a formal presentation, 
followed by small group activity - critical appraisal of an article 
relating to study design that was covered in the presentation. 
Therapy, harm, prognosis and diagnosis 

• Online learning 

Online lectures available on You Tube, which students had to watch 
before the respective 2 hour teaching block 

Resources delivered via Monash Library website 

• Mobile learning 

Incorporated in bedside teaching when students were interacting in 
the wards 

Access evidence related to specific patient 

Integrated learning, multifaceted intervention 

Collaborative learning, blended learning 

Asynchronous delivery 

• EBHC components 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the process 

Execution: 

• Module for third year medical students doing first clinical rotation 

• Ten 2 hour sessions 

Control (n= 234 ): Face-to-face learning (didactic learning) 

• Classroom activities 

• 2-hour teaching sessions: formal EBM concepts delivered by tutor/lecturer. 
Sessions commence with a formal presentation, followed by small group 
activity - critical appraisal of an article relating to study design that was 
covered in the presentation. Therapy, harm, prognosis and diagnosis 

Standalone learning, multifaceted intervention 

• EBHC components 

- General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the process 

Execution: 

• Module for third year medical students doing first clinical rotation 

• Ten 2 hour sessions 

Outcomes 

1. EBHC knowledge and skills 

ACE tool 

Berlin questionnaire 

One month post intervention 

2. EBHC attitude 

3. EBHC behaviour 

4. EBHC self-efficacy 
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Evidence-Based Practice Questionnaire (EBPQ) - self-reported 
measures related to implementation of EBM 

One month post intervention 

5. Satisfaction of students with method of learning 

Focus group discussions with selected participants 

6. Enablers of method of EBHC learning 

Focus group discussions with selected participants 

Notes 

• Country: Australia and Malaysia 

• Conflict of interest: Yes. DI coordinates the EBM program for the MBBS 
degree at Monash University. 

• Ethics approval: approval obtained 

• Funding: Australian Government Office for Learning and Teaching. 

• Authors contacted: yes - response obtained 

Over which period of time were the ten 2-hour EBM sessions 
delivered? 

Which baseline characteristics did you measure and do you have a 
table comparing these between groups? 

Did you adjust the results for clustering? 

How many groups were randomised to intervention/control? 


13 'iRisk of bias table 


Bias 

Authors 1 

[judgement 

Support for judgement 

Was the allocation sequence adequately 
generated? 

Yes 

Computerised random numbers 

Was the allocation adequately concealed? 

Yes 

Randomisation done by an independent 
researcher 

Were baseline outcome measurements 
similar? 

Unclear 

No pre-intervention measurement of outcomes 
done 

Were baseline characteristics similar? 

No 

No table of baseline characteristics present, no 
baseline characteristics measured (information 
supplied by author) 

Were incomplete outcome data 
adequately addressed? 

No 

Intervention group: 190/263 (72%) and in control 
group 160/234 (68%) of students did not 
complete the outcome assessment and were not 
analysed 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Yes 

Blinding of educators and students not possible; 
outcome assessors and analysts were blinded to 
allocation of participants 

Was the study adequately protected 
against contamination? 

Yes 

Cluster RCT 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other sources of bias identified 

Recruitment bias 

Unclear risk 



Not clear whether participants were first allocated 
to groups and groups were then randomised or 
vice versa 

Baseline imbalance 

High risk 

No table of baseline characteristics present 

Loss of clusters 

Unclear 

Not reported 

Incorrect analysis 

High risk 

Results not adjusted for clustering (information 
supplied by author) 

Compatibility with RCTs randomised by 
individuals 

High risk 

Results are not comparable to individually 
randomised RCTs since they were not adjusted for 
clustering 
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14. Kamin 2001 


Methods 

• RCT 

• Duration of study: Fall 1998 

• Setting: University of Colorado Health Sciences Centre 

• Learner context: Intervention group had additional training on using the 
software 

• Socio-economic context: high-income country 

Participants 

• Physician assistants, undergraduate 

• n=29 

• Inclusion criteria: 

2 nd year physician assistant students 

Interventions 

Intervention (n=?), e-learning: 

• EBM course with computer-mediated communication using asynchronous 
discussion software. Assignments, online discussions: weekly discussion 
topics; access to course website 

standalone EBHC learning, multifaceted intervention 

Pure e-learning, collaborative learning 

• EBHC components: 

Asking questions 

Accessing the literature 

Execution: 

• EBM course, during fall quarter of second year, according to individual needs 

• Learning theory: not reported 

• Delivery agent: Facilitator for discussions, otherwise self-directed 

Control (n=?): blended learning 

• EBM course with face-to-face discussions in mentor groups. Assignments and 
discussions: weekly discussion topics; Access to course website (details not 
described) 

standalone learning, multifaceted intervention 
blended learning, collaborative learning 

• EBHC components: 

Asking questions 

Accessing the literature 

Execution: 

• EBM course during fall quarter of second year, once weekly discussions. 

• Learning theory: not mentioned 

• Delivery agent: Instructor/facilitator for discussions; self-directed (website) 

Outcomes 

1. EBHC knowledge 

Measured with questionnaire 

Pre-and post-test. Pre-test was a subset of final examination (11 
questions requiring short answers) 

Notes 

• Country: USA 

• Conflict of interest: not reported 

• Ethics approval: not reported 

• Funding: University of Colorado Health Science Center Office of Education 
mini-grant program 

• Authors contacted: no 


i4.iRisk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Unclear 

Method of randomisation not explicitly described 

Was the allocation adequately 
concealed? 

Unclear 

not described 
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Were baseline outcome 
measurements similar? 

Unclear 

Baseline scores appear similar, but no p-values 
reported 

Intervention: mean 41 SD 12.4 

Control: mean 37 SD 12.3 

Were baseline characteristics 
similar? 

Unclear 

No table of baseline characteristics, “no significant 
differences existed between groups in any assessed 
parameter” 

Were incomplete outcome data 
adequately addressed? 

Unclear 

Not reported how many of the 27 participants were 
allocated to each group; flow of participants not 
reported 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Yes 

“Grader was blind to the identities of the group type 
until after the scores had been calculated” 

Was the study adequately protected 
against contamination? 

Unclear 

All students attended the same course, the only 
difference was the small group discussion sessions. 

Not described how contamination was prevented 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Unclear 

Very small sample size. 


15* Kok 2013 


Methods 

• Cluster RCT 

• Unit of randomisation: case-based learning groups 

• Duration of study: 12 months 

• Setting: Not reported where training took place 

• Socio-economic context: High-income country 

Participants 

• Medical doctors, continuing medical education 

• n=i32; 54 clusters 

• Inclusion criteria: 

Physicians belonging to a case-based learning groups from the 

Dutch National Institute of Benefit Schemes 

Interventions 

Intervention (n=67; 27 clusters): Blended learning: 

• Introductory, interactive e-learning course on EBM workshop with didactic 
and interactive sessions 

• Integrated learning, multifaceted learning 

• Blended learning, collaborative learning 

• Asynchronous delivery 

• EBHC components: 

General EBHC 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• EBM course with 5 contact days (full day) spread over 6 months 

• Learning theory: not mentioned 

• Delivery agent: Experts and teachers from the Dutch Cochrane Centre, the 
Netherlands School of Public and occupational health, the Coronel Institute 
of Occupational health and the Library of the Academic Medical Centre 

Control (n=65; 27 clusters) no learning: 

• Waiting list 

Outcomes 

1. EBHC knowledge and skills 

Adapted Fresno test, scores ranging from 0-212 

Measured at baseline, 7 months and 12 months 

2. EBHC attitude 

Likert scale 1-5 

Measured at baseline, 7 months and 12 months 
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3. EBHC behaviour 

Evidence-based disability evaluation - indicated by the 
frequency in use of evidence of sufficient quality in the 
disability evaluation reports. 6 quality indicators (max score: 

6): 

1) Presence of evidence 

2) Discernible EBM question 

3) Search strategy 

4) EBM source 

5) Evaluation of the quality of evidence 

Actual use of evidence in underpinning of the conclusion 
Measured 3 months after course (9 months after baseline) 

Notes 

• Country: The Netherlands 

• Conflict of interest: declared no conflict of interest 

• Ethics approval: exemption obtained 

• Funding: National institute of Benefit Schemes as part of the Research Centre 
for Insurance Medicine 

• Authors contacted: no 


15 'iRisk of bias table 


Bias 

1 1 

Authors 1 

[judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Computer generated list of random numbers 

Was the allocation adequately 
concealed? 

Yes 

Independent researcher was provided with a sequentially 
numbered list of the case-based learning groups and 
assigned the random sequence to this list. It was given to a 
research assistant. No changes to the list were allowed 

Were baseline outcome 
measurements similar? 

Yes 

Baseline outcomes similar for both groups 

Were baseline characteristics 
similar? 

Yes 

No baseline imbalances 

Were incomplete outcome data 
adequately addressed? 

No 

Loss to follow-up >20% in both groups, but similar for both 
groups: 

Intervention24% 

Control: 25% 

Was knowledge of the 
allocated interventions 
adequately prevented during 
the study? 

Yes 

Blinding of outcome assessors done 

Was the study adequately 
protected against 
contamination? 

Unclear 

Cluster RCT, but case-based learning groups, not described 
how they were kept from speaking to one another or where 
these groups were based 

Was the study free from 
selective outcome reporting? 

Yes 

all pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

no other sources of bias identified 

Recruitment bias 

Low risk 

Recruitment occurred before randomisation 

Baseline imbalance 

Low risk 

No baseline imbalances 

Loss of clusters 

jUnclear risk 

2 clusters (7%) lost in intervention group vs 0 clusters in 
control group 

Incorrect analysis 

Low risk 

! adjusted for clustering 

Compatibility with RCTs Low risk 

randomised by individuals 

Results similar to individually randomised studies 
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16. Kulier 2009 


Methods 

• Cluster RCT 

• Unit of randomisation: teaching hospitals 

• Study duration: August to December 2007 

• Setting: Clinical teaching hospitals in the UK and the Netherlands 

• Learner context: baseline EBHC knowledge similar btw groups 

• Socio-economic context: high-income countries 

Participants 

• Medical doctors, postgraduate 

• n=6i; 6 clusters 

• Inclusion criteria: 

Junior medical doctors who had not previously received formal EBM 
teaching in their postgraduate training 

Interventions 

Intervention (n=34; 3 clusters), blended learning 

• Clinically integrated EBM course with self-directed e-learning components 
and clinically relevant activities under the guidance of a facilitator 
(Knowledge needs identification in the clinical setting; independent study by 
using e-learning modules; interaction with facilitator throughout the course) 

• Integrated learning, multi-faceted intervention 

• Individual learning, blended learning 

• Asynchronous 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• 4-6 week self-directed course during postgraduate training 

• Learning theory: not mentioned 

• Delivery agent: self-directed, interaction with facilitator throughout course 
Control (n=36; 3 clusters), face-to-face learning 

• Lectures on EBM, using PowerPoint slides (same as in e-learning). 

Interaction with tutor during lectures 

• Standalone learning, single intervention 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

• Execution: 

4-6 week EBM course. Not reported how often and for how long 

• Learning theory: not reported 

• Delivery agent: Tutor 

Outcomes 

1. EBHC knowledge 

Measured with questionnaire with maximum of 62 points. Adapted 
from Berlin and Fresno test 

Pre-and post- intervention 

2. EBHC attitude 

Questions on 5 point Likert scale from strongly agree to strongly 
disagree 

Pre- and post-intervention 

Notes 

• Country: UK and The Netherlands 

• Conflict of interest: yes, none declared 

• Ethics approval: obtained 

• Funding: not reported 

• Authors contacted: no 
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i6.iRisk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence adequately 
generated? 

Yes 

Random sequence generated by computer 

Was the allocation adequately 
concealed? 

Unclear 

Not described 

Were baseline outcome measurements 
similar? 

Yes 

Figure 2 - no difference between baseline scores 

Were baseline characteristics similar? 

No 

No table of baseline characteristics and no mention 
of participant characteristics in text 

Were incomplete outcome data 
adequately addressed? 

No 

Loss to follow-up in intervention group: 8% 

Loss to follow-up in control group: 18% 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Unclear 

Not reported whether outcome assessors were 
blinded 

Was the study adequately protected 
against contamination? 

Yes 

Cluster randomisation. Participants did not rotate 
between clusters during the study period 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other source of bias identified 

Recruitment bias 

Unclear risk 

Recruitment of participants not described, not 
known whether randomisation occurred before or 
after recruitment 

Baseline imbalance 

[High risk 

Baseline characteristics not reported 

Loss of clusters 

Low risk 

Loss of clusters not reported, but max of 6 
participants lost to follow up 

Incorrect analysis 

Low risk 

Results adjusted for clustering 

Compatibility with RCTs randomised by 
individuals 

Low risk 

Results are similar to results from individually 
randomised RCTs 


17. Kulier 2012 


Methods 

• Cluster RCT 

• Unit of randomisation: Obstetrics and Gynaecological training units 

• Duration of study: April 2009 to November 2010 

• Setting: teaching hospitals in 7 LMICs 

• Learner context: access to internet and information were preconditions for 
being included in study 

• Socio-economic context: Low- and middle-income countries 

Participants 

• Medical doctors, postgraduate 

• n= 204 

• Inclusion criteria: 

To be eligible, the training unit had to be delivering EBM courses, 
defined as opportunities to learn about the techniques of EBM and its 
application in clinical practice, in the unit’s residency programme. 
Units had to have at least 4 residents who had not yet been exposed 
to formal EBM training and who were available for the duration of 
the trial to undertake the course and the assessment. They had to 
appoint a facilitator, a current clinical staff member knowledgeable 
about basic EBM principles to facilitate on the job training 
throughout the trial period. Appropriate computer equipment and 
access to relevant databases were a precondition. 

Interventions 

Intervention (n=i23; 31 clusters) blended learning: 
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Outcomes 

• Clinically integrated EBM e-learning course containing recorded 
presentations but incorporating learning activities, assignments and 
assessments in clinical practice. Clinical trainer involved in face-to-face 
teaching. 

• Integrated learning, multifaceted intervention 

• Blended learning, individual learning 

• Asynchronous delivery 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• Week course during obstetrics and gynaecology rotation, self-directed 
learning 

• Learning theory: learner-centred; just-in-time learning 

• Delivery agent: 

Clinical facilitator; self-directed 

Control (n=8i; 29 clusters), e- learning 

• Self-directed EBM teaching package containing recorded presentations. 

Access to facilitator that could be consulted on demand 

• Standalone learning; single intervention 

• Pure e-learning; individual learning 

• Asynchronous delivery 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

• Week course during obstetrics and gynaecology rotation, self-directed 
learning 

• Learning theory: not mentioned 

• Delivery agent: self-directed 

1. EBHC knowledge 

Measured with previously validated questionnaire; MCQs with a 
maximum of 62 points 

Pre-intervention and 4 weeks post intervention 

2. EBHC skills 

Evaluated using an objective structured clinical examination (score: 
0-14) 

Post-intervention only 

3. EBHC attitude 

Validated tool: 7 questions, using 5 point Likert scale from strongly 
agree to strongly disagree 

Pre-intervention (and post?) 

Notes 

• Country: 7 LMICs (Argentina, Brazil, Democratic Republic of Congo, India, 
Philippines, South Africa, Thailand) 

• Conflict of interest: yes, declared no conflicts of interest 

• Ethics approval: approval obtained 

• Funding: WHO 

• Authors contacted: Yes - missing data for attitude score post intervention. 


i7.iRisk of bias table 


| 

Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Computer-generated random numbers 
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1 

Was the allocation adequately 
concealed? 

Unclear 

not adequately described 

Were baseline outcome 
measurements similar? 

Yes 

Baseline knowledge scores not statistically different 

Were baseline characteristics 
similar? 

Yes 

No differences in baseline characteristics 

Were incomplete outcome data 
adequately addressed? 

Unclear 

Denominator for attrition rates in both groups is the 
number of participants that received the intervention. 
Number of participants randomised was not reported 
Intervention : 20% 

Control: 16% 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Unclear 

Not reported 

Was the study adequately 
protected against contamination? 

Yes 

Cluster RCT 

Was the study free from selective 
outcome reporting? 

Unclear 

For attitude, scores were only reported at baseline and 
not after the intervention 

Was the study free from other 
bias? 

| | 

Unclear 

Sample size of groups quite different 

Intervention: 123 

Control: 81 

Recruitment bias 

1 1 

|High risk 

Participants only consented to taking part in the study 
after randomisation 

Baseline imbalance 

Low risk 

No differences in baseline characteristics 

Loss of clusters 

[High risk 

7 clusters lost after randomisation per group 

Incorrect analysis 

Low risk 

[Adjusted for clustering 

Compatibility with RCTs 
randomised by individuals 

Low risk 

Results similar to individually randomised studies 


i8.Laibhen-Parkes 2014 


Methods 

• Cluster RCT 

• Allocation by units within hospital 

• Duration of study: 10 weeks 

• Setting: acute care, free-standing paediatric hospital in Southwestern USA 

• Learner context: 

• Socio-economic context: high-income countries 

Participants 

• Nurses, continuing nursing education 

• n= 58 (6 clusters) 

• Inclusion criteria: 

“Eligible participants for this study included BSN-prepared paediatric 
nurses who self-reported (a) having access to a computer that met the 
basic requirements for retrieving the modules, (b) having basic 
computer literacy skills, (c) having access to the Internet, (d) not 
working on the excluded campus of this paediatric hospital, (e) not 
participating in any formal EBP training program during the study 
timeline, and (f ) not having participated in the earlier version of the 
Web-based module piloted at this hospital” 

Exclusion criteria: 

“Nurses who worked on the excluded campus were not eligible to 
participate in this study because an earlier version of the EBP 
module had been piloted on this campus.” 

Interventions 

Intervention (n=33, 3 clusters) Pure e-learning: 

• Web-based modules on EBP with active links to instrumentation. 

• Stand-alone learning, single intervention 

• Pure e-learning, individual learning 
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• Asynchronous delivery 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying results 

Execution: 

2-hour module comprised of four units (20-30 minutes each) that 
were delivered via the Internet, in which participants were 
encouraged to review within 4 weeks 

• Learning theory: Novice to expert theory (cognitive change) and theory of 
planned behaviour (behaviour change) 

• Delivery agent: self-directed 

Control (n=25, 3 clusters): no EBHC learning 

• Attention control module 

Outcomes 

1. EBHC knowledge and skills 

Measured with adapted Fresno test 

Post-intervention only 

Pre-intervention and 4 weeks post intervention 

2. EBHC attitude (beliefs) 

Evidence-based Practice Beliefs (EBPB) scale 

Pre- and post-intervention 

Notes 

• Country: USA 

• Conflict of interest: yes, declared no conflicts of interest 

• Ethics approval: approval obtained 

• Funding: Jonas Nurse Leader’s Scholarship, Georgia Baptist College of 

Nursing that matched the funds from the Jonas Scholarship, and Nurse 

Faculty Loan Program 

• Authors contacted: Yes - response obtained 

How allocation of units was decided 

Whether outcome assessors were blind 

Whether results were adjusted for clustering 


18.1 Risk of bias 


Bias 

1 

Authors' 

judgement 

Support for judgement 



Was the allocation sequence 
adequately generated? 

Yes 

Used online randomisation tool (information provided 
by author) 

Was the allocation adequately 
concealed? 

Unclear 

Not clear whether allocation sequence was concealed 

Were baseline outcome 
measurements similar? 

No 

Significant difference between EBP beliefs at baseline. 
EBHC knowledge and skills only tested post- 
intervention. 

Were baseline characteristics 
similar? 


Table of baseline characteristics present, no significant 
differences between groups 

Were incomplete outcome data 
adequately addressed? 

No 

Lost to follow-up: 

Intervention: 58% (19/33) 

Control: 40% (10/25) 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Yes 

Outcome assessors were blind 

Was the study adequately 
protected against contamination? 

Unclear 

1 

Nurses from same hospitals were allocated to 
intervention/control. Although allocation was done per 
unit 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 
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Was the study free from other 
bias? 

Unclear 

Small sample size 

Recruitment bias 

Unclear 

Not clear whether participants were recruited before or 
after randomization 

Baseline imbalance 

Low risk 

No differences in baseline characteristics 

Loss of clusters 

Unclear 

Loss of clusters not reported 

Incorrect analysis 

High risk 

Not adjusted for clustering 

Compatibility with RCTs 
randomised by individuals 

High risk 

Results are not comparable to individually randomised 
RCTs since they were not adjusted for clustering 


19. Macrae 2004 


Methods 

• RCT 

• Duration of study: October 2001 to May 2002 (plus 6 weeks to complete 
exam) 

• Setting: home/hospital 

• Learner context: not described 

• Socio-economic context: high-income country 

Participants 

• Medical doctors (surgeons); continuing medical education 

• 11=83 

• Inclusion criteria: 

Members of the Canadian Association of General Surgeons with 
access to internet and email, need to agree to be randomised and to 
complete a written examination 

• Exclusion criteria: 

Surgeons with postgraduate training in clinical epidemiology 

Interventions 

Intervention (n=44): (e-learning) 

• Online journal club made up of 8 packages emailed to participants, each 
package containing one clinical and one methodological article. 

Asynchronous discussions on list serv, moderated by facilitator. Received 
methodological review and clinical review prepared by experts (methods and 
content respectively), also discussed on list serv. 

• Standalone learning, multifaceted intervention 

• Pure e-learning, collaborative learning 

• Asynchronous delivery 

• EBHC components: 

critical appraisal 

Execution: 

• Journal club over 8 months, 1 article discussed per month 

• List-serv discussion for 1 week on methodology; participants emailed 
methodological review and clinical review after which the list serv was open 
for another week 

• Learning theory: Adult learning theory 

• Delivery agent: 

Moderator for first week of list serv: general surgeon with 
training in clinical epidemiology 

Moderator for second week of list serv discussion: Surgeon with 
clinical epidemiology training and surgeon with expert 
knowledge in relevant content 

Control (n=37) e-learning: 

• Participants received 8 clinical articles per email and were given access to 
main medical and surgical journals some of which included articles on critical 
appraisal 

• Standalone learning, single intervention 

• Pure e-learning, individual learning 

• Asynchronous delivery 

• EBHC components: 
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Critical appraisal 

Execution: 

• Journal club over 8 months 

• Participants were sent reminders to read the article at the beginning of every 
month 

• Learning theory: no 

• Delivery agent: self-directed 

Outcomes 

1. EBHC skills (Critical appraisal skills) 

Post-test only: participants in both groups read 2 articles and 
completed rating scales on quality and gave free text responses to 
questions on methodology, validity and applicability of results. They 
had 6 weeks to complete this and could use any available resource 
(open-book) 

Notes 

• Country: Canada 

• Conflict of interest: not reported 

• Ethics approval: obtained 

• Funding: Educational grant from Ethicon and Ethicon Endo-surgery 

• Authors contacted: No 


lg.iRisk of bias table 


I I 

Bias 

Authors’ 

[judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Unclear 

Authors made use of blocked randomisation (blocks of 

10) but did not specify how sequence was generated 

Was the allocation adequately 
concealed? 

Unclear 

The complete list of assignments was sent to research 
assistant for implementation. No changes were made to 
the assignment 

Were baseline outcome 
measurements similar? 

Unclear 

No baseline assessment of outcomes 

Were baseline characteristics 
similar? 

Unclear 

Only looked at average number of years since graduation 
and whether they worked in the community or academic 
setting. There was no difference for these two variables 

Were incomplete outcome data 
adequately addressed? | 

No 

Loss to follow-up in intervention group: 42% 

Loss to follow-up in control group: 24% 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Yes 

Outcome assessors were blinded 

Was the study adequately 
protected against contamination? 

Unclear 

Although both interventions were internet-based, it is 
not clear where all the participants were based and 
whether they had the opportunity to discuss the articles. 1 

Was the study free from selective 
outcome reporting? 

Yes - 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other sources of bias identified 


20. McLeod 2010 


Methods 

• Cluster RCT 

• Unit of randomisation: general surgery training programmes 

• Duration of study: October 2008 to June 2009 

• Setting: Hospital/home, USA 

• Socio-economic context: high-income country 

Participants 

• Medical doctors (surgeons), continuing medical education 

• n= 44 i 

• Inclusion criteria: 
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General surgery training programmes in the United States that did 
not use Evidence based Reviews in Surgery (EBRS) packages in 
their journal club, with at least 10 residents in their programme 
who were agreeable to participating in the trial. 

Interventions 

Intervention (n= 225 ; 6 clusters) e-learning: 

• Online journal club made up of 8 packages emailed to participants, each 
package containing one clinical and one methodological article. 

Asynchronous discussions on list serv, moderated by facilitator. Clinical 
scenarios included in discussions 

• Standalone learning, single intervention 

• Pure e-learning, collaborative learning 

• Asynchronous delivery 

• EBHC components: 

Critical appraisal 

Execution: 

• Journal club over 8 months; l article package per month 

• During general surgery training 

• Self-directed 

• Learning theory: no 

• Delivery agent: 

Self-directed. Asynchronous discussions facilitated by 
methodological and clinical experts 

Control (n=2i6; 6 clusters) face-to-face learning: 

• Monthly face-to-face journal club using same articles, led by general surgical 
faculty member 

• Standalone learning, single intervention 

• EBHC components: 

Critical appraisal 

1 Execution: 

• Journal club over 8 months, once a month 

• during surgical residency 

• Learning theory: no 

• Delivery agent: General surgical faculty member 

Outcomes 

1. EBHC skills 

Critical appraisal test at designated examination room. Test 
consisted of 2 articles which participants had to read and then 
complete a series of short-answer questions and 7-point Likert 
scales to assess study quality. Score for each article was 48 (total 

96) 

Within one month of completion of EBRS (post-test only) 

2. Satisfaction of students with method of learning 

Likert scale questions from 1-5 (5= very satisfied) 

Notes 

• Country: USA 

• Conflict of interest: declared no conflicts of interest 

• Ethics approval: obtained 

• Funding: Physician Services incorporated 

• Authors contacted: No 


20.iRisk of bias table 


1 

Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Unclear 

Cluster randomisation done. Not described how 
sequence was generated 

Was the allocation adequately 
concealed? 

Unclear 

Not reported 

Were baseline outcome 
measurements similar? 

Unclear 

no measurement of baseline outcomes 
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Were baseline characteristics 
similar? 

Unclear 

No table of baseline characteristics and no 
description in text 

Were incomplete outcome data 
adequately addressed? 

No 

Intervention (internet) group loss to follow up: 42% 
Control (face-to-face) loss to follow up: 27% 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

No 

The critical appraisal test was marked by one of the 
investigators 

Was the study adequately protected 
against contamination? 

Yes 

Through cluster randomisation 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other bias? 

Yes 

No other sources of bias identified 

Recruitment bias 

Low risk 

Participants recruited before randomisation 

Baseline imbalance 

Unclear risk 

No table of baseline characteristics and no 
description in text 

Loss of clusters 

Unclear risk 

Loss to follow up assessed on an individual basis and 
loss of clusters not described 

Incorrect analysis 

1 

Low risk 

They mention that their analysis was “design 
adjusted” 

Compatibility with RCTs 
randomised by individuals 

Low risk 

Intervention effects are not much different from 
individually randomised RCTS 


21. Ramos-Morcillo 2015 


Methods 

• Non-RCT 

• Duration of study: not reported 

• Setting: Not well described - courses offered by the Nursing Council of Jaen 

• Socio-economic context: high income country 

Participants 

• Nurses, continuing medical education 

• n= 109 

• Inclusion criteria: 

Convenience sample of nursing professionals who attended free 
continuing education courses. Most from teaching hospitals and all 
had a Bachelor’s degree 

Interventions 

Intervention (n=54) Blended learning 

• EBP course consisting of 2 face-to-face sessions (5I1 each) and 30 hours of 
online learning - repository of learning material, exercises, discussion forum, 
consultations plus feedback 

• Stand-alone learning, multifaceted intervention 

• Blended learning, collaborative learning 

• Asynchronous delivery 

• EBHC components: 

Basic searching skills 

General EBHC 

Asking questions 

Accessing the literature 

Execution: 

• Not specified. 10 hours face-to-face and 30 hours online learning 

• Theory: Theory of planned behaviour 

• Delivery agent: 

Not reported 

Control (n= 55 ): no learning - course with non EBHC content 

Outcomes 

1. EBHC knowledge and skills 

2. EBHC attitude 

3. Practice 
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All outcomes measured with the Spanish version of the validated EBP 
questionnaire bbefore the course (01), 21 days after (02) and 60 days after (03). 
The questionnaire consists of 19 items, each item scored on a Likert scale from 1- 
7, a higher score indicating a more positive outcome 

Notes 

• Country: Spain 

• Conflict of interest: not reported 

• Ethics approval: obtained 

• Funding: Nursing Council of Jaen 

• Authors contacted: No 


2i.iRisk of bias table 


Bias 

Authors’ 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

No 

Non-randomised trial. Not described how participants 

were allocated 

Was the allocation adequately 

concealed? 

No 

Non-randomised trial 

Were baseline outcome 

measurements similar? 

Yes 

No significant differences between EBP knowledge & 
skills, attitude and practice at baseline. 

Were baseline characteristics 

similar? 

Yes 

See table 2. No significant differences btw baseline 

characteristics 

Were incomplete outcome data 
adequately addressed? 

Unclear 

Not reported 

Was knowledge of the allocated 
interventions adequately prevented 
during the study? 

Unclear 

Not reported 

Was the study adequately protected 
against contamination? 

Unclear 

Not clear. Nurses from various hospitals attended the 
course. Contamination can therefore not be completely 

excluded 

Was the study free from selective 
outcome reporting? 

Yes 

All outcomes reported in the methods section have 

been addressed in the results section 

Was the study free from other risks 

of bias? 

Yes 

No other bias identified 


22. Saunders 2016 


Methods 

• RCT 

• Duration of study: 12 months 

• Setting: University hospital system, Finland 

• Socio-economic context: high income country 

Participants 

• Nurses, continuing professional development 

• n= 85 

• Inclusion criteria: 
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Be a practicing RN at the university hospital system in any 
professional nursing role 

Work full- or part time as an RN in any unit of the university 
hospital system 

Be aged 21 or older and 

To be able to fluently read and understand Finnish 

These RNs were previously asked to complete a survey and asked to 
indicate whether they wanted to participate in an educational 
programme on EBP (n-379) 

Interventions 

Intervention (n=5o) blended learning 

Didactic EBP education session (4hours) plus EBP mentoring intervention and 
interactive e-learning module (4-hour live EBP education session, interactive 
clinical case situations for application of EBP into practice, interactive EBP review 
questions and answers) 

• integrated learning, multifaceted intervention 

• Blended learning, individual learning 

• Asynchronous delivery 

• EBHC components: 

General EBHC 

Accessing the literature 

Critical appraisal 

Applying the results 

Evaluating the process of EBP 

Execution: 

• 8 weeks course - once off 4 hour lecture and online learning 

• Theory: The Stevens Star Model of Knowledge Transformation and the 
advancing Research and clinical Practice through close collaboration model 
(ARCC) 

• Delivery agent: 

Advanced practice nurse 

Control (n=35): face-to-face learning on research utilisation 

• Although this is not defined as EBP - when looking at the content it is very 
similar to EBP (authors also point this out in the discussion) 

• Once off 4 hour didactic lecture 

• Content: 

Effective searching for evidence 

What is best available research evidence? 

Critical appraisal 

Summarising research evidence for decision-making 
Understanding, measuring and evaluating a research-based 
practice change 

Disseminating the results of a research-based practice 

Outcomes 

1. EBHC self-efficacy 

As part of the 

Measuring confidence in EBP 

2. EBHC Knowledge 

Measured as part of the Evidence-based Readiness inventory 

15 MCQ questions 
• Outcomes measured at 

To: pre-intervention 

Ti: Post-intervention (within 1 week) 

T2: within 8 weeks 

T3: 4 months after 

Notes 

• Country: Finland 

• Conflict of interest: none declared 

• Ethics approval: obtained 

• Funding: Finnish Nurses’ Education Foundation, Finnish Nurses’ 

Association, Saastamoinen Foundation, Finnish Work Environment Fund. 
Early Stage Researcher grant from University of Eastern Finland 
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• Authors contacted: Yes - to obtain means and SDs for knowledge outcomes. 


Awaiting response 


22.1 Risk of bias table 


Bias 

1 1 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Randomisation done by simple coin toss 

Was the allocation adequately 
concealed? 

|Unclear 

Not described 

Were baseline outcome 
measurements similar? 

[Unclear 

Not well reported - Figure 5. Contacted author, awaiting 
response 

Were baseline characteristics 
similar? 

No 

Proportion of nurse managers was significantly higher in 
intervention group (p=o.ooi) and the intervention group 
had more participants with a master’s degree (p=o.oo8) 

Were incomplete outcome data 
adequately addressed? 

High 

LTFU: 14% in intervention group vs 3% in control group. 
Reasons for LTFU in intervention group likely to be 
related to the outcome 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Unclear 

Not reported whether outcome assessors were blinded 

Was the study adequately 
protected against contamination? 

No 

Participants were from the same university hospital 
complex, educational interventions were delivered by the 
same APNs in control and intervention groups. Research 
utilisation is very similar to EBP 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other sources of bias identified 


1 

23. Schilling 2006 

Methods 

• Non-RCT 

• Duration of study: 21 months 

• Setting: Boston University, department of Family Medicine. Clerkship took 
place in community setting 

• Socio-economic context: high income country 

Participants 

• Medical doctors, undergraduate 

• n= 238 

• Inclusion criteria: 

3 rd year medical students enrolled in a clinical clerkship 

Interventions 

1 

Intervention (n=i34) e-learning: 

• Web-based modules on searching and selecting best evidence and calculating 
NNT within Family Medicine clerkship 

• Integrated learning, multifaceted intervention 

• Pure e-learning, collaborative learning 

• Asynchronous delivery 

• EBHC components: 

Epidemiology 

Basic searching skills 

Asking questions 

Accessing the literature 

Applying the results 

Execution: 
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• 6 weeks module 

• According to individual needs, 40-60 minutes for the first two weeks (content 
only) 

• 3rd year medical students in Family Medicine Clerkship 

• Learning theory: Adult learning concepts and the utility of reflective learning 
and collaborative or peer-driven learning 

• Delivery agent: 

Faculty moderated discussions 

Control (n=i 04 ): no learning 

Outcomes 

1. EBHC skills 

Analysis of students’ MEDLINE search strategies (measured by 
Librarians with Likert scale) 

Analysis of retrieved articles identified as providing best evidence 
to address a clinical case study (measured by Librarians with 

Likert scale) 

NNT test (self-reported) 

Pre-and post-survey on students’ perceptions regarding their 
skills 

During 6 th week, students were given clinical case and had to 
search and select best article (objectively measured) 

2. EBHC attitude 

Measured with Likert scale from strongly disagree to strongly 
agree 

Pre-and post-survey 

Notes 

• Country: USA 

• Conflict of interest: not reported 

• Ethics approval: obtained 

• Funding: The Robert Wood Johnson Foundation 

• Authors contacted: No 


23*1 Risk of bias table 


Bias 

Authors 1 

[judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

No 

Alternate blocks of clerks were assigned to the 
intervention group 

Was the allocation adequately 
concealed? 

No 

Alternate blocks of clerks were assigned to the 
intervention groups 

Were baseline outcome 
measurements similar? 

Unclear 

Performance outcomes only measured after course 

Were baseline characteristics 
similar? 

Yes 

No table of baseline characteristics, but adequate 
description in paragraph 

Were incomplete outcome data 
adequately addressed? 

Unclear 

No flow diagram of participants. Not all blocks allocated 
to intervention and control groups received all the 
questionnaires. Overall attrition: 14.5% (not reported per 
group) 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Yes 

Librarians who graded the searching skills were blinded 
to the group allocation 

Was the study adequately 
protected against contamination? 

[Unclear 

Students attending same university, and possibly have 
contact to other students in different rotations 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other sources of bias identified 
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24-Welch 2014 


Methods 

• RCT 

• International study (all members of the National Athletic Trainer Association) 

• Countries of origin of participants and socio-economic contexts not reported. 

Participants 

• Athletic trainers (“health care professionals who collaborate with physicians. 
The services provided by ATs comprise prevention, emergency care, clinical 
diagnosis, therapeutic intervention and rehabilitation of injuries and medical 
conditions” 

http://www.nata.0r2/sites/default/files/Athletic Trainer Profile.pdf ) 

• Undergraduate, postgraduate and CME 

• n= 473 

• Inclusion criteria: 

All members of the National Athletic Trainer Association (NATA) 
were eligible to participate 

Interventions 

Intervention (n= 237 ) Pure e-learning: 

• Web-based modules on EBP 

• Standalone learning, single intervention 

• Pure e-learning, individual learning 

• Asynchronous delivery 

• EBHC components: 

Asking questions 

Accessing the literature 

Critical appraisal 

Applying the results 

Execution: 

4 weeks module 

According to individual needs, 10 learning modules lasting 25 minutes each 

• Learning theory: not mentioned 

Control (n=i 04 ): no learning 

Outcomes 

l.EBHC Knowledge 

Web-based assessment developed by the research team, 
consisting of multiple-choice questions on each module 

Maximum of 60 marks 

Pre-and post-test 

Notes 

• Country: various 

• Conflict of interest: not reported 

• Ethics approval: obtained 

• Funding: NATA board of directors 

• Authors contacted: No 


24.1 Risk of bias table 


Bias 

Authors 1 

judgement 

Support for judgement 

Was the allocation sequence 
adequately generated? 

Yes 

Using a random number generator (SPSS) 

Was the allocation adequately 
concealed? 

Unclear 

Participants were sent individualised emails informing 
them which group they belonged to. Not reported who 
sent the emails and whether they had access to the 
randomisation sequence. 

Were baseline outcome 
measurements similar? 

Yes 

Baseline score intervention: mean 30.99 SD 5.93 

Baseline score control: mean 30.12 SD 5.73 

Were baseline characteristics 
similar? 

Unclear 

Table of baseline characteristics present, but without p- 
values 
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Were incomplete outcome data 
adequately addressed? 

No 

Attrition rates very high in both groups: 

Intervention: 63% (149/237) 

Control:63% (149/236) 

Was knowledge of the allocated 
interventions adequately 
prevented during the study? 

Unclear 

Not reported whether outcome assessors were blinded 

Was the study adequately 
protected against contamination? 

Unclear 

Not clear where participants were from. NATA is an 
international network, so it is assumed that participants 
were from all over the world but it is also possible that 
most of them were from one country or the same 
institution. Not clear whether they have opportunities to 
contact each other. 

Was the study free from selective 
outcome reporting? 

Yes 

All pre-specified outcomes reported on 

Was the study free from other 
bias? 

Yes 

No other sources of bias identified 


Return to text 


9.4 CHARACTERISTICS OF ONGOING STUDIES 


1. Schneider 2015 


Study name 

DELIVER 

Methods 

This study has two parts: 1) Online national survey on EBHC attitudes, skills 
and use of evidence in clinical practice 

2) a prospective randomised wait-list controlled trial 

Participants 

Chiropractors in the US 

Interventions 

Intervention: 

- structured online educational module on EBP 

Control: 

- wait list 

Outcomes 

EBHC skills and attitudes 

Starting date 

? 

Contact information 

Michael Schneider: mjs@pitt.edu 

Notes 

The study was identified through a conference abstract. The author was 
contacted. The first phase of this study has been published (baseline survey) 
while the data for the randomised trial is currently being analysed. 


Return to text 


9.5 CHARACTERISTICS OF EXCLUDED STUDIES 


Study ID 

Reason for exclusion 

Ahmadi 2011 

Intervention not eligible: learning content not EBHC 

Allen 2015 

Study design not eligible: No control group 

Amsallem 2007 

Intervention not eligible: learning content not EBHC 

Badgett 2001 

Intervention not eligible: not delivered via e-learning 

Bell 2008 

Intervention not eligible: learning content not EBHC 

Boyd 2015 

Study design not eligible: No control group, descriptive study 

Bradley 2002 

Intervention not eligible: not delivered via e-learning 

Buchanan 2014 

Intervention not eligible: not delivered via e-learning 

Casebeer 2008 

Intervention not eligible: learning content not EBHC 

Grad 2005 

Intervention not eligible: not delivered via e-learning 
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Gupta 2014 

Study design not eligible: descriptive study, not experimental 

Hammond 2006 

Study design not eligible: no control group, this is a descriptive study 

Johnston 2004 

Study design not eligible: cross-sectional study 

Kobt 2015 

Study design not eligible: No control group 

Leathers 2013 

Intervention not eligible: Training on using website containing practice 
guidelines, not EBHC knowledge and skills 

Leung 2003 

Intervention not eligible: not delivered via e-learning 

Long 2016 

Outcomes not eligible: EBHC outcomes were only measured in the 
intervention group 

Lewis 2007 

Study design not eligible: descriptive study, not experimental 

Maloney 2015 

Intervention not eligible: Learning content not EBHC 

Mary 2013 

Study design not eligible: descriptive study, not experimental 

Murtaugh 2005 

Intervention not eligible: learning content not EBHC 

Nelson 2007 

Intervention not eligible: learning content not EBHC 

Phillips 2015 

Intervention not eligible: Learning content not EBHC 

Qureshi 2015 

Study design not eligible: No control group 

Ruzafa-Martinez 2016 

Intervention not eligible: not delivered via e-learning 

Schifferdecker 2008 

Intervention not eligible: learning content not EBHC 

Stewart 2005 

Intervention not eligible: learning content not EBHC 

Webber 2010 

Study design not eligible: no experimental study 

Whittaker 2015 

Intervention not eligible: Learning content not EBHC 

Widayahening 2012 

Intervention not eligible: not delivered via e-learning 


Return to text 
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9.6 SUMMARY OF CHARACTERISTICS OF INCLUDED STUDIES 


9.6.1 Pure e-learning vs no learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Laibhen- 

Parkes 

2014 

USA 

58 

Cluster 

RCT 

Nurses 

Web -based module on 
EBP 

Attention control 
module (no 

EBHC) 

• EBHC 
knowledge 

• EBHC skills 

• EBHC attitude 

Shilling 

2006 

USA 

134 

Non- 

RCT 

Undergraduate 

medical 

students 

Web -based modules on 
searching and selecting 
best evidence and 
calculating NNT within 
Family Medicine 
clerkship 

Traditionally 
structured Family 
Medicine 
clerkship, no 
modules on EBM 

• Searching skills 

• Attitude 

Welch 

2014 

Not 

reported 

473 

RCT 

Athletic 

trainers 

Web -based module on 
EBP 

No learning 
(waiting list) 

• EBHC 
knowledge 


9.6.2 Blended learning vs no learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Bergold 

2013 

Germany 

120 

RCT 

Junior doctors 

Online EBM course 
consisting of 
presentations and 
exercises. EBM tutor was 
available for questions on 
content and technology 

No learning 
(waiting list) 

• EBM 
knowledge 

• Usefulness of 
EBM course 

Dizon 

2014 

Philippines 

54 

RCT 

Physiotherapists 

EBP training (face-to- 
face) plus EBP checklist 
and online support 

No learning 
(waiting list) 

• EBM 
knowledge 

• EBM skills 

• EBM attitude 

• EBM behaviour 

Forsetlund 

2003 

Norway 

148 

RCT 

Public health 
physicians 

EBM workshop with 
access to databases and 
participation in 
asynchronous discussion 
list 

Access to 
databases for 
one year (no 
learning) 

• EBP behaviour 

• EBP attitudes 

• EBP knowledge 

Kok 2013 

Netherlands 

132 (54 
clusters) 

Cluster 

RCT 

Physicians 

Blended learning: 
introductory, interactive 
e-learning course on EBM 
workshop with didactic 
and interactive sessions 

No learning 

• EBM behaviour 

• EBM 
knowledge 

• EBM skills 

• Self-efficacy 

Ramos- 

Morcillo 

2015 

Spain 

109 

Non- 

RCT 

Nurses 

EBP course: two face-to- 
face sessions with 
additional online learning 

No learning 
(course on 
different topic) 

• EBP knowledge 
and skills 

• EBP attitude 

• EBP practice 
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9*6*3 Pure e-learning vs face-to-face learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Bradley 

2005 

Norway 

175 

RCT 

Undergraduate 

medical 

students 

Computer-assisted 
(CD-ROM) on five 
steps of EBM 

Workshop on five 
steps of EBM 

• EBM knowledge 

• EBM skills 

• EBM attitudes 

Davis 2007 

UK 

55 

RCT 

Medical 

interns 

Short, computer- 
based session on 

EBM (CD-ROM) 

Lecture on EBM with 
similar content 
structure and 
duration as 
intervention 

• EBM knowledge 

• EBM attitude 

Davis 

2008 

UK 

229 

RCT 

Undergraduate 

medical 

students 

Short, computer- 
based session on 

EBM (CD-ROM) 

Lecture on EBM with 
similar content 
structure and 
duration as 
intervention 

• EBM knowledge 

• EBM attitude 

Hadley 

2010 

UK 

237 (7 

clusters) 

Cluster 

RCT 

Medical 

interns 

Clinically integrated 
teaching of EBM: 
Three modules on 
EBM (asking 
questions, accessing 
literature, appraising 
literature). Unlimited 
access for six weeks 

Standalone, three 
hour face-to-face 
lecture (same content 
as intervention) 

• EBM knowledge 

• EBM skills 

Horiuchi 

2009 

Japan 

93 

RCT 

Nurses 

E-learning of EBM 
divided into four 
parts, distributed 
according to 
individual progress 

EBM teaching 
divided into one 
evening lecture per 
week (for one month) 

• EBM knowledge 

• Satisfaction with 
learning 

McLeod 

2010 

USA 

441 (12 
clusters) 

Cluster 

RCT 

Surgical 

residents 

Online journal club 
made up of 8 
packages emailed to 
participants, each 
package containing 
one clinical and one 
methodological 
article. 

Asynchronous 
discussions on list 
serv, moderated by 
facilitator. Clinical 
scenarios included in 
discussions 

Monthly face-to-face 
journal club using 
same articles, led by 
general surgical 
faculty member. 

• Critical appraisal 

skills 
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9*6*4 Blended learning vs face-to-face learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Brettle 

2013 

UK 

77 

RCT 

Nurses (pre- 
registration 
diploma) 

Online tutorial on 
searching within the 
“Foundations in Nursing” 
module 

Lecture on 
searching (same 
content as online 
tutorial) within 
the “Foundations 
in Nursing” 
module 

• Searching skills 

Ilic 2013 

Australia 

61 

Non- 

RCT 

Undergraduate 

medical 

students 

Blended learning 
consisting of a face-to- 
face workshop, self- 
directed learning via the 
Monash University’s 
website, presentations of 
patient-based EBM cases 

Didactic 

lectures / tutorials 
with small group 
tasks and 
discussions 

• EBM knowledge 

• EBM skills 

• EBM attitude 

Ilic 2015 

Australia 

and 

Malaysia 

497 

Cluster 

RCT 

Undergraduate 

medical 

students 

Blended learning 
consisting of classroom 
activities (formal lectures 
and small group 
activities), online lectures 
on You Tube, access to 
resources on the library 
website and mobile 
access to the evidence at 
the bedside. 

Face-to-face 
learning 
consisting of 
classroom 
activities (formal 
lectures and 
small group 
activities) 

• EBM knowledge 
and skills 

• EBM attitude 

• EBM attitude 

• EBM self- 
efficacy 

• Satisfaction 
with learning 

• Enablers of 
method of 
learning 

Kulier 

2009 

UK and 
Netherlands 

61 (6 
clusters) 

Cluster 

RCT 

Obstetrics and 
gynaecology 
residents 

Clinically integrated EBM 
course with self-directed 
e-learning components 
and clinically relevant 
activities 

Lectures on 

EBM, using 
PowerPoint 
slides (same as 
in e-learning). 
Interaction with 
tutor during 
lectures 

• EBM knowledge 

• EBM attitude 

Saunders 

2016 

Finland 

85 

RCT 

Nurses 

Educational intervention 
on EBP consisting of 
didactic lectures, 
interactive online EBP 
module and EBP 
mentorship 

Didactic lectures 
on research 
utilisation (very 
similar content 
to EBP) 

• EBP self- 
efficacy 

• EBP knowledge 
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9.6.5 Blended learning vs pure e-learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Fernandez 

2014 

Australia 
and Hong 
Kong 

187 

Non- 

RCT 

Postgraduate 

nursing 

students 

Evidence-based 

practice (EBP) 

DVD: 

demonstrations of 

EBP skills provided 

on a DVD 

1) Standard distance 
method 

2) Computer Lab 
teaching method (on 
campus) - practical 
interactive computer- 
based training with 
immediate feedback 

3) Face to face didactic 
classroom teaching 
method - classroom 
teaching 

• EBP 
knowledge 

• EBP skills 

Kamin 

2001 

USA 

2 7 

RCT 

Undergraduate 

physician 

assistants 

Online EBM course 
with computer- 
mediated 
communication 
using 

asynchronous 
discussion software 

Online EBM course with 
face-to-face discussions in 
mentor groups 

• EBM 

knowledge 

Kulier 

2012 

Argentina, 
Brazil, 
Democratic 
Republic of 
Congo, 
India, 

Philippines, 

South 

Africa, 

Thailand 

204 (60 
clusters) 

Cluster 

RCT 

Obstetrics and 
gynaecology 
residents 

Clinically 
integrated EBM e- 
learning course 
containing 
recorded 
presentations but 
incorporating 
learning activities, 
assignments and 
assessments in 
clinical practice. 
Clinical trainer 
involved in face-to- 
face teaching. 

Self-directed EBM teaching 
package containing 
recorded presentations. 
Access to facilitator that 
could be consulted on 
demand. 

• EBM 
knowledge 

• EBM skills 

• EBM 
attitudes 

• Educational 
environment 


9.6.6 Pure e-learning vs pure e-learning 


Study ID 

Country 

Sample 
size (n) 

Study 

design 

Participants 

Intervention 

Comparison 

Main outcomes 

Brouwers 

2011 

Canada 

87 

RCT 

Healthcare 
professionals, 
Clinicians, 
methodologists, 
policy makers 
and trainees 

1) Online tutorial on 
AGREEII tool 

2) Online tutorial on 
AGREE II tool plus 
practice feedback 

Participants were 
given the AGREE 
II manual 

• EBM skills 
(appraising a 
guideline) 

• Satisfaction with 
learning 

• Self-efficacy 

• Attitudes 

MacRae 

2004 

Canada 

81 

RCT 

Surgeons 

Online journal club 
made up of 8 packages 
emailed to participants, 
each package containing 
one clinical and one 
methodological article. 
Asynchronous 
discussions on list serv, 
moderated by facilitator 

Participants also 
received 8 
packages per 
email and were 
given access to 
main medical 
and surgical 
journals. 

• Critical appraisal 

skills 


119 


The Campbell Collaboration | www.campbellcollaboration.org 





9-9 SUMMARY OF RESULTS 



Comparison 

Pure e- 
learning 
vs 

no 

learning 

Blended 
learning 
vs no 
learning 

Pure e- 
learning vs 
face-to- 
face 

learning 

Blended 
learning 
vs face-to-face 
learning 

Blended 
learning 
vs pure e- 
learning 

Pure e- 
learning 
vs pure e- 
learning 

Outcome 

SMD (95% Confidence Intervals) 

EBHC 

knowledge 

0.71 (0.40 
to 1.01) 

1 study, 

n=i 75 

0.50 (0.13 
to 0.86) 

1 study, 
11=119 

-0.03 (-0.26 
to 0.20) 

5 studies, 
11=632 

0.28 (-0.23 to 
0.79) 

1 study, n=i46 

0.69 (0.40 to 

0.99) 

2 studies, 
n=i 93 

not reported 

EBHC 
knowledge 
& skills 

0.47 (-0.27 
to 1.21) 

1 study, 
11=29 

1.40 (-0.06 
to 2.85) 

2 studies, 
11=163 

not reported 

-0.23 (-0.52 to 
0.06) 

2 studies, n=i84 

not reported 

not reported 

EBHC 
knowledge 
& skills (1 
month) 

not reported 

0.90 (0.42 
to 1.38) 

2 studies, 
11=241 

not reported 

not reported 

not reported 

not reported 

EBHC 
knowledge 
& skills 
( 3 + 

months) 

not reported 

1.11 (0.80 
to 1.42) 

2 studies, 
11=186 

not reported 

not reported 

not reported 

not reported 

EBHC 

skills 

Not 

reported 

not 

reported 

-0.15 (-0.34 
to 0.04) 

2 studies, 
n=457 

-0.21 (-0.68 to 
0.26) 

1 study, n=70 

-0.53 (-2.31 
to 1.25) 

2 studies, 
n=2i8 

1.30 (0.68 
to 1.93) 

2 studies, 
11=119 

EBHC 

attitude 

1.05 (0.26 
to 1.83) 

1 study, 
11=29 

0.17 (-0.09 
to 0.43) 

2 studies, 
11=226 

0.11 (-0.27 to 
0.48) 

3 studies, 
11=111 

1.07 (0.57 to 1.58) 

1 study, n=69 

not reported 

not reported 

EBHC 
attitude 
(1 month) 

not reported 

0.05 (-0.34 
to 0.44) 

2 studies, 
11=241 

not reported 

not reported 

not reported 

not reported 

EBHC 

attitude 

( 3 + 

months) 

not reported 

0.32 (-0.02 
to 0.67) 

1 study, 
n=i32 

not reported 

not reported 

not reported 

not reported 

EBHC 

behaviour 

not reported 

0.06 
(-0.28 to 
0.40) 

3 studies, 

11 = 207 

not reported 

2.34 (1.72 to 2.96) 

1 study, n=69 

not reported 

not reported 

EBHC 
behaviour 
(1 month) 

not reported 

0.19 (-0.19 
to 0.56) 

1 study, 
11=109 

not reported 

not reported 

not reported 

not reported 

EBHC 

behaviour 

( 3 + 

months) 

not reported 

0.61 (0.21 
to 1.01) 

1 study, 
11=100 

not reported 

not reported 

not reported 

not reported 
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io Figures 


Figure 5: Summary of risk of bias across all included studies 


Was the allocation sequence adequately generated? 

Was the allocation adequately concealed? 
Were baseline outcome measurements similar? 

Were baseline characteristics similar? 
Were incomplete outcome data adequately adressed? 
Was knowledge of the allocated interventions adequately prevented during the study? 

Was the study adequately protected against contamination? 
Was the study free from selective outcome reporting? 

Was the study free from other bias? 

Recruitment bias 
Baseline imbalance 
Loss of clusters 
Incorrect analysis 
Compatibility with RCTs randomised by individuals 



■ Low risk of bias 


1 | Unclear risk of bias 


U High risk of bias 


Return to text 
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Figure 6: Risk of bias judgement per included study 
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ii Data and analyses 


Abbreviations related to type of healthcare professionals and level of education: 

• AT: Athletic trainers 

• CPD: Continuing Professional Development 

• M: Medical doctors 

• MIX: Mix of various healthcare professionals at different levels of education 

• N: Nurses 

• PG: Postgraduate 

• PT: Physiotherapists 

• UG: Undergraduate 


11.1 PURE E-LEARNING VS NO LEARNING 


11.1.1 Analysis l.i: EBHC knowledge 


Study or Subgroup Std. Mean Difference SE 

Experimental 

Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Random, 95% Cl 

Std. Mean Difference Risk of Bias 

IV, Random, 95% Cl ABCDEFGH 1 JKLMN 

1.1.1 RCT 

Welch 201 4: AT CPD 0.7072 0.1559 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 4.54 (P < 0.00001) 

88 

88 

87 

87 

100.0% 

100.0% 

0.71 [0.40,1.01] 
0.71 [0.40, 1.01] 



Total (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 4.54 (P < 0.00001) 

Test for subgroup differences: Not applicable 

Risk of bias leaend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

88 

87 

100.0% 

0.71 [0.40,1.01] 







-1 -0.5 ( 

Favours no learning 

0*5 J 

Favours pure e-learning 


(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 
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n.1.2 Analysis 1.2: EBHC knowledge and skills 


Experimental Control Std. Mean Difference Std. Mean Difference Risk of Bias 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl IV, Random, 95% Cl ABCDEFGH I JKLMN 

1.2.1 RCT 

Laibhen-Parkes 2014: N_CPD (1) 0.4708 0.3775 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 
Test for overall effect: Z= 1 .25 (P = 0.21) 

Total (95% Cl) 

Heterogeneity: Not applicable 
Test for overall effect: Z= 1 .25 (P = 0.21) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) not adjusted for clustering. 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


14 15 100.0% 

14 15 100.0% 


0.47 [-0.27,1.21] 
0.47 [-0.27,1.21] 


0.47 [-0.27,1.21] 


-+- 


• ? •#•#?•??• ? •• 


-+ 


Favours no learning Favours Pure e-learning 


Risk of bias leoend 

(A) Was the allocation sequence adequately generated? 


Return to text 


11.1.3 Analysis 1.3: EBHC attitude 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

1.3.1 RCT 

Laibhen-Parkes 201 4: N_CPD 1.0482 0.4 14 

Subtotal (95% Cl) 14 

Heterogeneity: Not applicable 
Test for overall effect: Z= 2.62 (P = 0.009) 

Total (95% Cl) 14 

Heterogeneity: Not applicable 
Test for overall effect: Z= 2.62 (P = 0.009) 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


15 100.0% 
15 100.0% 


1.05 [0.26, 1.83] 
1.05 [0.26, 1.83] 


1.05 [0.26, 1.83] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


• + ••••*• + $•<? •• 



Favours no learning Favours pure e-leaming 


Return to text 


124 


The Campbell Collaboration | www.campbellcollaboration.org 


11.1.4 Analysis 1 . 4 : Attrition of learners 


Study or Subgroup 

E-learning 
Events Total 

No learning 
Events Total 

Weight 

Risk Ratio 

M-HI, Fixed, 95% Cl 

Risk Ratio 

M-H, Fixed, 9 5% CII 

Risk of Bias 

ARCDEFGH 1 J K L M N 

1.4,2 RCT 










Laibhen-Parkes 201 4: N_CPD 

19 

33 

10 

25 

7.1% 

1.44(0.82,2.53] 



• ? •••• ?•??•?•• 

Welch 201 4: AT CPD 

149 

237 

149 

236 

92.9% 

1.00(0.87,1.1 4] 

■ 

••••••••• 

Subtotal {95% Cl) 


270 


261 

100.0% 

1.03 [0.90, 1.1-S] 




Total events 

168 


159 








Heterogeneity: Chi 3 = 1.58, df= 1 (P= 0.21); F= 37% 

Test for overall effect: Z= 0.39 (P = 0.70) 

1 1 1 1 

0.5 0.7 1 1.5 2 

Favours e-learning Favours no learning 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

[D> Were baseline characteristics similar? 

(E)Were incomplete outcome data adequately adressed? 

{F} Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

{1} Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

[MJ Compatibility with RCTs randomised by individuals 


Return to text 
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11.2 BLENDED LEARNING VS NO LEARNING 


11.2.1 Analysis 2.1: EBHC knowledge 


Study or Subgroup Std. Mean Difference 

SE 

Experimental 

Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Random, 95% Cl 

Std. Mean Difference 

IV, Random, 95% Cl A B C D 

Risk of Bias 

EFGH 1 JKLMN 

2.1.1 RCT 

Forsetlund 2003: M CPD(1) 0.4968 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z = 2.67 (P = 0.008) 

0.1863 

58 

58 

61 

61 

100.0% 

100.0% 

0.50 [0.1 3, 0.86] 
0.50 [0.1 3, 0.86] 


t 


Total (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z = 2.67 (P = 0.008) 

Test for subgroup differences: Not applicable 


58 

61 

100.0% 

0.50 [0.1 3, 0.86] 


♦ 







-2 -\ ( 

Favours no learning 

\ 2 

Favours blended learning 



Footnotes Risk of bias legend 

(1) Concept knowledge (A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 


11.2.2 Analysis 2.2: EBHC knowledge and skills 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random. 95% Cl 

2.2.1 Non-RCT 


Ramos-Morcillo 2015: N_CPD 0.6801 0.1972 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 3.45 (P = 0.0006) 


54 55 51.9% 0.68 [0.29. 1 .07] 

54 55 51.9% 0.68 [0.29, 1.07] 


Std. Mean Difference 
IV, Random, 95% Cl 


♦ 


Risk of Bias 

ABCDEFGH I JKLMN 


2.2.2 RCT 

Dizon 201 4: PT_CPD 2.1652 0.3477 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 6.23 (P < 0.00001) 


27 27 48.1 % 

27 27 48.1% 


2.1 7 [1.48, 
2.17 [1.48, 


2.85] 

2.85] 


Total (95% Cl) 81 82 100.0% 

Heterogeneity: Tau a = 1.02; Chi a = 13.80, df= 1 (P = 0.0002); l a = 93% 

Test for overall effect: Z= 1.88 (P = 0.06) 

Test for subgroup differences: Chi a = 13.80, df= 1 (P= 0.0002), l a = 92.8% 

Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 



Favours no learning Favours blended learning 


Return to text 
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n. 2.3 Analysis 2.3: EBHC knowledge and skills (1 month post-intervention) 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

2.3.1 Non-RCT 


Ramos-Morcillo 201 5: N_CPD (1) 0.651 4 0.1 968 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 3.31 (P = 0.0009) 


54 55 49.3% 0.65 [0.27, 1 .04] 

54 55 49.3% 0.65 [0.27, 1.04] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


2.3.2 RCT 

Kok2013: M_CPD 1.1424 0.1882 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 6.07 (P < 0.00001) 

Total (95% Cl) 

Heterogeneity: Tau 2 = 0.08; Chi== 3.25, df= 1 (P = 0.07); l 2 = 69% 

Test for overall effect: Z= 3.67 (P = 0.0002) 

Test for subgroup differences: Chi 2 = 3.25, df= 1 (P = 0.07), l 2 = 69.2% 
Footnotes 

(1) 60 days after start of training 


67 65 

67 65 


121 120 

4 4 ] i 

Favours no learning Favours blended learning 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


50.7% 

50.7% 


1.1 4 [0.77, 1.51] 
1.14 [0.77, 1.51] 


0.90 [0.42, 1.38] 


11.2.4 Analysis 2.4: EBHC knowledge and skills (3+months post-intervention) 


Study or Subgroup Std. Mean Difference SE 

Experimental 

Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

2.4.1 RCT 

Dizon 201 4: PT_CPD (1) 1.7599 0.3241 

Kok 201 3: M CPD (2) 0.9006 0.183 

Subtotal (95% Cl) 

Heterogeneity: Chi 2 = 5.33, df= 1 (P = 0.02); l 2 = 81 % 

Test for overall effect: Z= 6.96 (P < 0.00001) 

27 

67 

94 

27 

65 

92 

24.2% 

75.8% 

100.0% 

1.76 [1.1 2, 2.40] 
0.90 [0.54, 1.26] 

1.11 [0.80,1.42] 


♦ 

###+•#++» 

Total (95% Cl) 

Heterogeneity: Chi 2 = 5.33, df= 1 (P = 0.02); l 2 = 81 % 

Test for overall effect: Z= 6.96 (P < 0.00001) 

Test for subgroup differences: Not applicable 

94 

92 

100.0% 

1.11 [0.80,1.42] 








-2 -1 ( 
Favours no learning 

i 2 

Favours blended learning 


Footnotes Risk of bias legend 

(1) 3 months post training (A) Was the allocation sequence adequately generated? 

(2) 6 months post training (B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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n.2.5 Analysis 2.5: EBHC attitude 


Experimental 

Study or Subgroup Std. Mean Difference SE Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Random, 95% Cl 

Std. Mean Difference Risk of Bias 

IV, Random, 95% Cl ABCDEFGH 1 JKLMN 

2.5.1 Non-RCT 

Ramos-Morcillo 2015: N CPD 0.2226 0.1922 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 1.16 (P = 0.25) 

54 

54 

55 

55 

48.1% 

48.1% 

0.22 [-0.1 5, 0.60] 
0.22 [-0.15, 0.60] 





2.5.2 RCT 

Forsetlund 2003: M CPD 0.1201 0.1852 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.65 (P = 0.52) 

56 

56 

61 

61 

51.9% 

51.9% 

0.1 2 [-0.24, 0.48] 
0.12 [-0.24, 0.48] 



Total (95% Cl) 

Heterogeneity: Tau 2 = 0.00; Chi== 0.15, df= 1 (P = 0.70); l 2 = 0% 

Test for overall effect: Z = 1 .27 (P = 0.20) 

Test for subgroup differences: Chi 2 = 0.15, df= 1 (P = 0.70), l 2 = 0% 

110 

116 

100.0% 

0.17 [-0.09, 0.43] 







-\ -0.5 ( 

Favours no learning 

o's 1 

Favours blended learning 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


11.2.6 Analysis 2.6: EBHC attitude (one month post-intervention) 


Experimental 

Study or Subgroup Std. Mean Difference SE Total 

Control 

Total 

Weight 

Std. Mean Difference 
IV, Random, 95% Cl 

Std. Mean Difference 

IV, Random, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

2.6.1 Non-RCT 

Ramos-Morcillo 201 5: N CPD(1) -0.1565 0.1919 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.82 (P= 0.41) 

54 

54 

55 

55 

48.0% 

48.0% 

-0.1 6 [-0.53, 0.22] 
-0.16[-0.53,0.22] 




1 



2.6.2 RCT 








Kok 201 3: M CPD 0.2423 0.1748 

67 

65 

52.0% 

0.24 [-0.1 0,0.58] 




Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 1 .39 (P = 0.1 7) 

67 

65 

52.0% 

0.24 [-0.10, 0.58] 




Total (95% Cl) 

Heterogeneity: Tau 2 = 0.05; Chi 2 = 2.36, df= 1 (P = 0.1 2); l 2 = 58% 

Test for overall effect: Z= 0.25 (P = 0.80) 

Test for subgroup differences: Chi 2 = 2.36, df= 1 (P = 0.1 2), l 2 = 57.6% 
Footnotes 

121 

120 

100.0% 

0.05 [-0.34, 0.44] 

— 1 1 

-2 -1 ( 
Favours no learning 

Risk of bias leaend 

1 1 — 

1 2 

Favours blended learning 


(1) 60 days after start of training (A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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11 . 2.7 


Analysis 2.7: EBHC attitude (3+ months post-intervention) 


Experimental Control 

Study or Subgroup Std. Mean Difference SE Total Total 

2.7.2 RCT 

Kok2013:M_CPD(1) 0.3231 0.1753 67 65 

Subtotal (95% Cl) 67 65 

Heterogeneity: Not applicable 
Test for overall effect: Z = 1 .84 (P = 0.07) 


Std. Mean Difference 
Weight IV, Random, 95% Cl 


100.0% 0.32 [-0.02, 0.67] 

100.0% 0.32 [-0.02, 0.67] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


Total (95% Cl) 67 65 100.0% 

Heterogeneity: Not applicable 

Test for overall effect: Z = 1 .84 (P = 0.07) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) 6 months after training 



Favours no learning Favours blended learning 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 


11.2.8 Analysis 2.8: EBHC behaviour 




Experimental 

Control 


Std. Mean Difference 


Std. Mean Difference 



Risk of Bias 

Study or Subgroup Std. Mean Difference 

SE 

Total 

Total 

Weight 

IV, Random, 95% Cl 


IV, Random, 95% Cl 


A B C D 

EFGH 1 JKLMN 

2.8.1 Non-RCT 

Ramos-Morcillo 201 5: N CPD(1) -0.1079 

0.1917 

54 

55 

51.8% 

-0.11 [-0.48,0.27] 







Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.56 (P= 0.57) 


54 

55 

51.8% 

-0.11 [-0.48,0.27] 







2.8.2 RCT 

Forsetlund 2003: M CPD (2) 0.2377 

0.2028 

50 

48 

48.2% 

0.24 [-0.1 6, 0.64] 







Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z = 1 .1 7 (P = 0.24) 


50 

48 

48.2% 

0.24 [-0.16, 0.64] 







Total (95% Cl) 

Heterogeneity: Tau 3 = 0.02; Chi 3 = 1 .53, df= 1 (P = 0.22); l 3 = 

35% 

104 

103 

100.0% 

0.06 [-0.28, 0.40] 











-2 

-1 ( 

1 

2 




Test for overall effect: Z= 0.34 (P = 0.73) 

Test for subgroup differences: Chi== 1 .53, df= 1 (P = 0.22), l 3 = 34.8% 


Favours no learning Favours blended learning 


Footnotes 

(1) EBPQ- practice 

(2) Hypothetical assignment 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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11,2.9 Analysis 2.9: EBHC behaviour (one month post-intervention) 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

2.9.1 Non-RCT 


Ramos-Morcillo 2015: N_CPD 0.1873 0.192 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.98 (P = 0.33) 


54 55 100.0% 0.1 9 [-0.1 9, 0.56] 

54 55 100.0% 0.19 [-0.19, 0.56] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


»• 


Total (95% Cl) 54 55 100.0% 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.98 (P = 0.33) 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


0.19[-0.19, 0.56] 


Favours no learning Favours blended learning 


11.2.10 Analysis 2.10: EBHC behaviour (3+ months post-intervention) 


Study or Subgroup Std. Mean Difference 

SE 

Experimental 

Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Random, 95% Cl 

Std. Mean Difference 

IV, Random, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

2.10.1 RCT 

Kok2013: M CPD (1) 0.6087 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 2.97 (P = 0.003) 

0.2048 

51 

51 

49 

49 

100.0% 

100.0% 

0.61 [0.21,1.01] 

0.61 [0.21,1.01] 


* 


Total (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 2.97 (P = 0.003) 

Test for subgroup differences: Not applicable 


51 

49 

100.0% 

0.61 [0.21,1.01] 


♦ 






-2 -i ( 

Favours blended learning 

i 2 

Favours no learning 


Footnotes Risk of bias legend 

(1) Evidence-based disability evaluations (A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 
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11,2,11 Analysis 2.11: Attrition of learners 


E-learning No learning Risk Ratio 


Study or Subgroup 

Events 

Total 

Events 

Total 

Weight 

M-H, Random, 95% CJ 

2.11.1 RCT 

Bergold 2013: M_CPD 

2 

60 

4 

60 

11.7% 

0.50 [0.10, 2.63] 

Kok 201 3: M CRD 

16 

67 

16 

65 

88.3% 

0.97 [0.53, 1.77] 

Subtotal (95% Cl) 

Total events 

18 

127 

20 

125 

100.0% 

0.90 [0.51, 1.58] 

Heterogeneity: Tau 3 = 0.00; Chi 3 = 

0.55, df 

= 1 (P = 

0.46); I 3 

: = 0% 


Test for overall effect: Z = 

0.37 (P = 

: 0.71) 






Risk Ratio 

M-H, Random, 95% Cl 

. 


Risk of Bias 

ABCDEFGH i J K L M N 


•••••••••••••• 


Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

{F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

01) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


0.02 0.1 


10 


50 


Favours e-learning Favours no learning 


Return to text 


11.3 PURE E-LEARNING VS FACE-TO-FACE LEARNING 


11.3.1 Analysis 3.1: EBHC knowledge 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

3.1.1 RCT 


Bradley 2005: M_UG 


0.0306 

0.1543 

83 

85 

24.3% 

0.03 [-0.27, 0.33] 

Davis 2007: M_CPD 


0.4804 

0.2749 

25 

30 

12.6% 

0.48 [-0.06, 1.02] 

Davis 2008: MJJG 


-0.3312 

0.1542 

70 

109 

24.3% 

-0.33 [-0.63, -0.03] 

Hadley 201 0: M_CPD (1 ) 


0.0238 

0.1589 

88 

72 

23.7% 

0.02 [-0.29, 0.34] 

Horiuchi 2009: N CPD 


-0.1573 

0.2398 

37 

33 

15.1% 

-0.1 6 [-0.63, 0.31] 

Subtotal (95% Cl) 




303 

329 

100.0% 

-0.03 [-0.26, 0.20] 

Heterogeneity: Tau a = 0.03; Chi a = 

7.74, df: 

= 4(P = 

0.10); l a = 48% 





Test for overall effect: Z= 0.27 (P = 

: 0.79) 







Total (95% Cl) 




303 

329 

100.0% 

-0.03 [-0.26, 0.20] 


Heterogeneity: Tau a = 0.03; Chi a = 7.74, df= 4 (P = 0.1 0); l a = 48% 
Test for overall effect: Z= 0.27 (P= 0.79) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) Adjusted for clustering and baseline score 


Std. Mean Difference Risk of Bias 

IV, Random, 95% Cl ABCDEFGH I JKLMN 





-1 -0.5 

0 5 1 


Favours face:face leamin Favours pure e-learning 


(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 
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n.3.2 Analysis 3.2 EBHC skills 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

3.2.1 RCT 


Bradley 2005: M_UG (1 ) 

-0.0302 0.1535 

82 

88 

38.0% 

-0.03 [-0.33,0.27] 

McLeod 2010: M_PG 

-0.2284 0.119 

130 

157 

62.0% 

-0.23 [-0.46, 0.00] 

Subtotal (95% Cl) 

Heterogeneity: Tau 2 = 0.00; Chi 2 = 1 .04, df = 1 (P = 0.31 ); I 2 = 4% 
Test for overall effect: Z = 1 .59 (P = 0.1 1 ) 

212 

245 

100.0% 

-0.15[-0.34,0.04] 

Total (95% Cl) 

Heterogeneity: Tau 2 = 0.00; Chi 2 = 1.04, df= 1 (P = 0.31); l 2 = 4% 

212 

245 

100.0% 

-0.15[-0.34,0.04] 


Test for overall effect: Z= 1.59 (P= 0.11) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) Critical appraisal skills 


Std. Mean Difference Risk of Bias 

IV, Random, 95% Cl ABCDEFGH I JKLMN 





1 1 1 1 

-1 -0.5 0.5 1 

Favours face:face learnin Favours pure e-learning 

Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 


11.3.3 Analysis 3.3: EBHC attitude 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

3.3.1 RCT 


Bradley 2005: M_UG 0.1062 0.19 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.56 (P= 0.58) 


57 54 100.0% 0.11 [-0.27,0.48] 

57 54 100.0% 0.11 [-0.27,0.48] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


#•••••#•# 


Total (95% Cl) 57 54 100.0% 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.56 (P= 0.58) 

Test for subgroup differences: Not applicable 
Risk of bias leoend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


0.11 [-0.27,0.48] 



Favours face-to-face Favours pure e-learning 


Return to text 
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11.3.4 Analysis 3.4: Attrition of learners 



E4earning 

Face-to-face learning 


Risk Ratio 

Study or Subgroup 

Events Total 

Events 

Total 

Weight 

M-H, Random, 95% Cl 

3.4.1 RCT 

Bradley 2005: MJJG 

2 85 

5 

90 

11.5% 

0.42 [0.08,2.12] 

Davis 2008: MJJG 

44 114 

6 

115 

19.5% 

7.40 [3.28, 16.67] 

Hadley 2010: M_CPD 

34 122 

43 

115 

24.1% 

0.75 [0.51,1.08] 

Horiuchi 2009: N_CPD 

8 45 

15 

48 

20.2% 

0.57 [0.27,1.21] 

McLeod 201 0: M PG 

95 225 

59 

216 

24.8% 

1.55 [1.18,2.02] 

Subtotal (95% Cl) 

591 


584 

100.0% 

1.24 [0.59, 2.59] 

Total events 

183 

128 




Heterogeneity: Tau a = 0.55; Chi a = 35.20, df= 4 (P < 0.00001); l a = 89% 


Test for overall effect: Z = 

0.57 (P = 0.57) 





Total (95% Cl) 

591 


584 

100.0% 

1.24 [0.59, 2.59] 

Total events 

183 

128 





Heterogeneity: Tau a = 0.55; Chi a = 35.20, df= 4 (P < 0.00001); l a = 89% 

Test for overall effect: Z= 0.57 (P = 0.57) 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Risk Ratio 

M-H, Random, 95% Cl 


Risk of Bias 

ABCDEFGH IJKLMN 


*???#? •#•••••• 

*•??#??#• 

? ? ? ? ••••••??•• 


H 

0.005 


— h- 

0.1 


10 


— h 

200 


Favours e-learning Favours face-to-face 


Return to text 


11.4 BLENDED LEARNING VS FACE-TO-FACE LEARNING 


11.4.1 Analysis 4.1: EBHC knowledge 




Experimental 

Control 


Std. Mean Difference 

Std. Mean Difference 

Risk of Bias 

Study or Subgroup Std. Mean Difference 

SE 

Total 

Total 

Weight 

IV, Random, 95% Cl 

IV, Random, 95% Cl 

ABCDEFGH 1 JKLMN 

4.1.1 RCT 

Kulier 2009: M PG 0.2807 

0.2583 

28 

33 

100.0% 

0.28 [-0.23, 0.79] 


■ 


Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 1.09 (P = 0.28) 


28 

33 

100.0% 

0.28 [-0.23, 0.79] 




Total (95% Cl) 

Heterogeneity: Not applicable 


28 

33 

100.0% 

0.28 [-0.23, 0.79] 









- 2 -\ 1 

1 1 

I 1 2 


Test for overall effect: Z = 1 .09 (P = 0.28) 

Test for subgroup differences: Not applicable 






Favours face:face leamin 

Favours blended learning 



Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 
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11.4.2 Analysis 4.2: EBHC knowledge and skills 



Experimental 

Control 


Std. Mean Difference 

Std. Mean Difference 

Risk of Bias 

Study or Subgroup Std. Mean Difference SE 

Total 

Total 

Weight 

IV, Random, 95% Cl 

IV, Random, 95% Cl 

ABCDEFGH 1 JKLMN 

4.2.1 Non-RCT 

llic 201 3: M UG (1) -0.238 0.2587 

Subtotal (95% Cl) 

34 

34 

27 

27 

32.9% 

32.9% 

-0.24 [-0.75, 0.27] 

-0.24 [-0.75, 0.27] 






Heterogeneity: Not applicable 

Test for overall effect: Z= 0.92 (P = 0.36) 





4.2.2 RCT 

llic 201 5: M UG (2) -0.2319 0.181 

61 

62 

67.1% 

-0.23 [-0.59, 0.1 2] 




Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 1.28 (P = 0.20) 

61 

62 

67.1% 

-0.23 [-0.59, 0.12] 




Total (95% Cl) 

95 

89 

100.0% 

-0.23 [-0.52, 0.06] 




Meierogeneiry: 1 au - u.uu; uni - u.uu, ar= 1 = u.yu;; 1 - u% 




-2 -1 

i 2 


Test for overall effect: Z= 1.58 (P= 0.11) 

Test for subgroup differences: Chi a = 0.00, df= 1 (P = 0.98), l a = 0% 
Footnotes 

(1) Berlin tool 

(2) Berlin tool - adjusted for clustering with DE of 1.199 (ICC 0.01) 


Favours face:face learnin Favours blended learning 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 


11.4.3 Analysis 4.3: EBHC skills 


Study or Subgroup Std. Mean Difference 

SE 

Experimental 

Total 

Control 

Total 

Weight 

Std. Mean Difference 

IV, Random, 95% Cl 

Std. Mean Difference 

IV, Random, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

4.3.1 RCT 

Brettle 201 3: N UG -0.2147 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 0.90 (P = 0.37) 

0.2398 

35 

35 

35 

35 

100.0% 

100.0% 

-0.21 [-0.68, 0.26] 
-0.21 [-0.68, 0.26] 

’ 



Total (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z = 0.90 (P = 0.37) 

Test for subgroup differences: Not applicable 


35 

35 

100.0% 

-0.21 [-0.68, 0.26] 


r , , 







, , 

-2 -i ( 

Favours face:face learnin 

i \ 2 

Favours blended learning 



Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 
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ii. 4<4 Analysis 4.4: EBHC attitude 


Experimental Control 

Study or Subgroup Std. Mean Difference SE Total Total 

4.4.1 RCT 

llic 201 5: M_UG (1) 1.0741 0.2591 37 32 

Subtotal (95% Cl) 37 32 

Heterogeneity: Not applicable 
Test for overall effect: Z = 4.1 5 (P < 0.0001 ) 


Std. Mean Difference 
Weight IV, Random, 95% Cl 


100.0% 1.07(0.57,1.58] 

100.0% 1.07(0.57,1.58] 


Std. Mean Difference Risk of Bias 

IV, Random, 95% Cl ABCDEFGH I JKLMN 



Total (95% Cl) 

Heterogeneity: Not applicable 
Test for overall effect: Z = 4.1 5 (P < 0.0001 ) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) adjusted for clustering with DE of 1.199 (ICC 0.01) 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


1.07(0.57,1.58] 


- 2-10 1 2 
Favours face-to-face Favours nlended learning 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 


Return to text 


11.4.5 Analysis 4.5: EBHC behaviour 


Study or Subgroup 
4.5.1 RCT 
llic 201 5: M_UG (1) 
Subtotal (95% Cl) 


Experimental Control Std. Mean Difference 

Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 


2.3428 0.317 


37 32 100.0% 2.34(1.72,2.961 

37 32 100.0% 2.34(1.72,2.96] 


Heterogeneity: Not applicable 

Test for overall effect: Z= 7.39 (P < 0.00001) 


Std. Mean Difference 
IV, Random, 95% Cl 


Risk of Bias 

ABCDEFGH I JKLMN 


Total (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 7.39 (P < 0.00001) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) adjusted for clustering with DE of 1.199 (ICC 0.01) 


37 32 100.0% 


2.34(1.72,2.96] 

— r-\ — i — 1 — 1 — 

Favours face-to-face Favours blended learning 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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11.4*6 Analysis 4.6: Attrition of learners 


E-leaming Face-to-face learning Risk Ratio Risk Ratio 

Study or Subgroup Events Total Events Total Weight M-H, Random, 95% Cl M-H, Random, 95% Cl 

4.6.1 Non-RCT 


llic 201 3: M UG 0 34 

0 

27 

Not estimable 

Subtotal (95% Cl) 0 


0 

Not estimable 

Total events 0 

0 



Heterogeneity: Not applicable 




Test for overall effect: Not applicable 





Risk of Bias 

ABCDEFGH I JKLMN 




4.6.3 RCT 


Brettle 201 3: N JJG 

5 

40 2 

37 

13.0% 

2.31 [0.48,11.20] 

llic 201 5: M_UG 

190 

263 160 

234 

61.2% 

1.06 [0.94, 1.19] 

Kulier 2009: M_PG 

6 

34 3 

36 

17.4% 

2.1 2 [0.57, 7.80] 

Saunders 2016: N CPD 

7 

50 1 

35 

8.4% 

4.90 [0.63, 38.07] 

Subtotal (95% Cl) 


387 

342 

100.0% 

1.50 [0.79, 2.85] 

Total events 

208 

166 




Heterogeneity: Tau 2 = 0.1 7; Chi 2 = 4.59, df= 3 (P = 0.20); l 2 = 

35% 



Test for overall effect: Z = 

1.25 (P= 0.21) 




Total (95% Cl) 


387 

342 

100.0% 

1.50 [0.79, 2.85] 

Total events 

208 

166 





Heterogeneity: Tau 2 = 0.1 7; Chi 2 = 4.59, df= 3 (P = 0.20); l 2 = 35% 

Test for overall effect: Z = 1 .25 (P = 0.21) 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


000000000 
00700 * 000 *##00 
•••••••••••••• 

0 ? ? 0 ? ? •00 


Favours e-learning Favours face-to-face 
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11.5 BLENDED LEARNING VS PURE E-LEARNING 


11.5.1 Analysis 5.1: EBHC knowledge 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

5.1.1 RCT 


Kamin 2001: PAJJG 

0.3386 0.3884 

14 

13 

15.0% 

0.34 [-0.42, 1.10] 

Kulier 201 2: M PG 

0.7544 0.1633 

98 

68 

85.0% 

0.75 [0.43, 1.07] 

Subtotal (95% Cl) 


112 

81 

100.0% 

0.69 [0.40, 0.99] 

Heterogeneity: Tau 2 = 0.00; Chi= = 

= 0.97, df= 1 (P= 0.32); l 2 = 0% 





Test for overall effect: Z= 4.60 (P 

< 0.00001) 





Total (95% Cl) 


112 

81 

100.0% 

0.69 [0.40, 0.99] 


Heterogeneity: Tau 2 = 0.00; Chi 2 = 0.97, df= 1 (P = 0.32); l 2 = 0% 

Test for overall effect: Z= 4.60 (P < 0.00001) 

Test for subgroup differences: Not applicable 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Std. Mean Difference Risk of Bias 

IV, Random, 95* Cl ABCDEFGH I JKLMN 


* 



Favours pure e-leaming Favours blended learning 
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11.5.2 Analysis 5.2: EBHC skills 


Experimental Control Std. Mean Difference 

Study or Subgroup Std. Mean Difference SE Total Total Weight IV, Random, 95% Cl 

5.2.1 Non RCT 


Fernandez 201 4: N_PG(1) -1.4614 0.3156 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 4.63 (P < 0.00001) 


24 28 48.9% -1 .46 [-2.08, -0.84] 

24 28 48.9% -1 .46 [-2.08, -0.84] 


Std. Mean Difference 
IV, Random, 95% Cl 



Risk of Bias 

ABCDEFGH I JKLMN 


? 


5.2.2 RCT 

Kulier 201 2: M_PG 0.3592 0.1591 98 

Subtotal (95% Cl) 98 

Heterogeneity: Not applicable 
Test for overall effect: Z= 2.26 (P = 0.02) 

Total (95% Cl) 122 

Heterogeneity: Tau a = 1 .59; Chi a = 26.53, df= 1 (P < 0.00001); l a = 96% 

Test for overall effect: Z= 0.58 (P = 0.56) 

Test for subgroup differences: Chi a = 26.53, df= 1 (P < 0.00001), l a = 96.2% 
Footnotes 

(1) Critical appraisal skills: Computer lab vs. DVD 


68 51.1% 

68 51.1% 


96 100.0% 

SS — i — 1 — 5 

Favours pure e-learning Favours blended learning 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


0.36 [0.05, 0.67] 
0.36 [0.05, 0.67] 


-0.53[-2.31,1.25] 


11.5.3 Analysis 5.3: EBHC skills (additional results) 


Study or Subgroup Std. Mean Difference 

SE 

Experimental 

Total 

Control 

Total 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

5.3.1 Formulating PICO question 









Fernandez 201 4: N_PG (1) 

-0.2619 

0.2795 

24 

28 

-0.26 [-0.81, 0.29] 

1— 



Fernandez 201 4: N_PG (2) 

-1.2373 

0.2281 

99 

28 

-1.24 [-1.68, -0.79] 

1 


••???•?•? 

5.3.2 Searching skills 









Fernandez 201 4: N_PG (3) 

0.4882 

0.2826 

24 

28 

0.49 [-0.07, 1.04] 


1 

••? »+##•# 

Fernandez 201 4: N_PG (4) 

-0.2304 

0.2145 

99 

28 

-0.23 [-0.65, 0.19] 

1- 


••???•?•? 

5.3.3 Level of evidence 









Fernandez 201 4: N_PG (5) 

-0.8524 

0.2914 

24 

28 

-0.85 [-1.42, -0.28] 

1 


••???#?+? 

Fernandez 201 4: N_PG (6) 

-1.4803 

0.2339 

99 

28 

-1.48 [-1.94, -1.02] 

1 

1 

1 

1 

•#???•?•? 

1 


- 2-1012 
Favours pure e-learning Favours blended learning 


Footnotes 

(1) Computer lab vs. DVD 

(2) Didactic lectures vs. DVD 

(3) Computer lab vs. DVD 

(4) Didactic lectures vs. DVD 

(5) Computer lab vs. DVD 

(6) Didactic lectures vs. DVD 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(I) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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11.5.4 Analysis 5.4: Attrition of learners 


Favours e-learning Control Risk Ratio Risk Ratio RiskofBias 

Study or Subgroup Events Total Events Total Weight P.1 -H, Fixed, 95% Cl M-H, Fixed, 95% Cl ABCDEFGH 1 JKLMN 

5.4.1 RCT 

Kulier 201 2: M_PG (1) 25 123 

Subtotal (95% Cl} 123 

Total events 25 

Heterogeneity: Not applicable 
Test for overall effect: Z = 0.76 (P = 0.45) 

Total {95% Cl} 123 

Total events 25 

Heterogeneity: Not applicable 
Test for overall effect: Z= 0.76 (P = 0.45) 

Test for subgroup differences: Not applicable 
Footnotes Risk of bias legend 

{1)The denominatorfor both grops is the number of participants that received the intervention.. (A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the... 

(G) Was the study adequately protected against contamination? 

[H}Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


13 


13 


81 100.0% 1.27(0.69,2.33] 
81 100.0% 1.27 [0.69, 2.33] 


81 100.0% 1.27 [0,09, 2.33] 


0.01 0.1 


? ## ? ? • ? ? 


10 100 


Favours intervention Favours control 


Return to text 


11.6 PURE E-LEARNING VS PURE E-LEARNING 


11.6.1 Analysis 6.1: EBHC skills 




Experimental 

Control 


Std. Mean Difference 

Std. Mean Difference 

Risk of Bias 

Study or Subgroup Std. Mean Difference 

SE 

Total 

Total 

Weight 

IV, Random, 95% Cl 

IV, Random, 95% Cl 

ABCDEFGH 1 JKLMN 

6.1.1 Non-RCT 

Fernandez 201 4: N PG(1) 

1.6275 

0.2925 

28 

36 

49.6% 

1.63 [1.05, 2.20] 



••••••••• 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 5.56 (P < 0.00001) 



28 

36 

49.6% 

1.63 [1.05, 2.20] 




6.1.2 RCT 

Macrae 2004: M CPD (2) 

0.9858 

0.2872 

26 

29 

50.4% 

0.99 [0.42,1.55] 



7 7 7 7 •• 7 •• 

Subtotal (95% Cl) 

Heterogeneity: Not applicable 

Test for overall effect: Z= 3.43 (P= 0.0006) 



26 

29 

50.4% 

0.99 [0.42, 1.55] 




Total (95% Cl) 

Heterogeneity: Tau 2 = 0.12; Chi 2 = 2.45, df= 



54 

65 

100.0% 

1.30 [0.68, 1.93] 


♦ 


1 = u. 1 zj, r= 





-2 -1 ( 

1 i 2 


Test for overall effect: Z= 4.07 (P < 0.0001) 

Test for subgroup differences: Chi 3 = 2.45, df= 1 (P= 0.12), l 2 = 59.2% 

Footnotes 

(1) Critical appraisal: DVD vs standard distance 

(2) Electronic journal club with asynchronous discussion group vs. receiving articles via email and access to journals 


Favours control Favours intervention 
Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the... 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 
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11.6.2 Analysis 6.2: EBHC skills (additional results) 


Study or Subgroup 

Std. Mean Difference 

SE 

Experimental 

Total 

Control 

Total 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Std. Mean Difference 

IV, Fixed, 95% Cl 

Risk of Bias 

ABCDEFGH 1 JKLMN 

6.2.1 PICO 









Fernandez 201 4: N_PG 

0.7998 

0.2623 

28 

36 

0.80 [0.29,1.31] 


— 1 — 


6.2.2 Searching skills 

0.2439 

0.253 

28 

36 

0.24 [-0.25, 0.74] 




Fernandez 201 4: N_PG 

6.2.3 Levels of evidence 




Fernandez 201 4: N_PG 

1.2788 

0.2777 

28 

36 

1.28 [0.73, 1.82] 

— 1 1 

— 1 

1 1— 

••••••••• 


Favours standard distance Favours DVD 


(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the study? 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

0) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 

(M) Incorrect analysis 

(N) Compatibility with RCTs randomised by individuals 


Return to text 


11.6.3 Analysis 6.3: Attrition of learners 


Favours e-learning Control Risk Ratio 

Study or Subgroup Events Total Events Total Weight M-H, Random, 95% Cl 

6,3.1 RCT 


Risk Ratio 

M-H, Random, 95% Cl 


Risk of Bias 

ABCDEFGH 1 JKLMN 


Brouwers 201 1_MIX (1) 

9 

29 

4 

14 

23.5% 

1.09 

[0.40, 2.92] 

Brouwers 201 1_MIX (2) 

10 

30 

4 

14 

24.5% 

1.17 

[0.44, 3.08] 

Macrae 2004: M CPD 

19 

45 

9 

38 

52.1% 

1.78 

[0.92, 3.47] 

Subtotal (95% Cl) 


104 


66 

100.0% 

1.43 

[0.89, 2.31] 

Total events 

38 


17 





Heterogeneity: Tau 3 = 0.00; Chi 3 = 

0.89, df= 

= 2 (P = 

: 0.64); l 3 = 

0% 




Test for overall effect: Z= 1 .46 (P : 

= 0.14) 







Total (95% CII} 


104 


66 

100.0% 

1.43 

[0.89, 2.31] 

Total events 

38 


17 





Heterogeneity: Tau 3 = 0.00; Chi 3 = 

0.89, df= 

= 2 (P = 

: 0.64); l 3 = 

0% 





Test for overall effect: Z= 1 .46 (P = 0.1 4) 

Test for subgroup differences: Not applicable 
Footnotes 

(1) Group 1 

(2) Group 2 


I ? ? •• ? ••• 
+??##? ••• 
? ? ? ? • 4 ? #• 


I 1 

0.01 0.1 
Favours intervention 


1 

10 100 
Favours control 


I 


Risk of bias legend 

(A) Was the allocation sequence adequately generated? 

(B) Was the allocation adequately concealed? 

(C) Were baseline outcome measurements similar? 

(D) Were baseline characteristics similar? 

(E) Were incomplete outcome data adequately adressed? 

(F) Was knowledge of the allocated interventions adequately prevented during the... 

(G) Was the study adequately protected against contamination? 

(H) Was the study free from selective outcome reporting? 

(!) Was the study free from other bias? 

(J) Recruitment bias 

(K) Baseline imbalance 

(L) Loss of clusters 
(M.) Incorrect analysis 

[N> Compatibility with RCTs randomised by individuals 


Return to text 


139 


The Campbell Collaboration | www.campbellcollaboration.org 


12 


Summary of findings 


Abbreviations: 

CPD: Continuing professional development 
UG: Undergraduate 
PG: Postgraduate 


12.1 PURE E-LEARNING VS NO LEARNING (3 STUDIES) 


Outcome 

SMD 

(95% 

Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

knowledge 

0.71 
(040 to 
1.01) 

n=i 75 
(1 study) 

Web-based 
module on 
EBP 

Athletic 
trainers 
(CPD); country 
unknown 

High risk for 
attrition bias, 
unclear risk of 
selection and 
detection bias 
Imprecision: size of 
the effect ranges 
from small to large 

EBHC 
knowledge 
and skills 

0.47 
(-0.27 to 
1.21) 

n=29 
(1 study) 

Web-based 
module on 
EBP 

Nurses (CPD); 
USA 

High risk for 
selection and 
attrition bias 

Results not 
adjusted for 
clustering 

Small sample size 
and imprecision 

EBHC 

skills 

MD 0.8 

(p<0.05) 

n=i34 
(1 study) 

Web-based 
module on 
searching 

Medical 
doctors (UG); 
USA 

High risk of 
selection bias 

Authors did not 
report SDs or 

95 %CI 

Small sample size 

EBHC 

attitude 

1.05 
(0.26 to 
1-83) 

n=29 
(1 study) 

Web-based 
module on 
EBP 

Nurses (CPD), 
USA 

High risk for 
selection and 
attrition bias 

Results not 
adjusted for 
clustering 

Small sample size 
and imprecision: 
effect ranges from 
small to very large 
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12.2 BLENDED LEARNING VS NO LEARNING (5 STUDIES) 


Outcome 

SMD 
( 95 % Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

knowledge 

0.50 
(0.13 to 
0.86) 

n=ii9 

1 study 

- EBM 
workshop 
with access 
to databases 
and 

asynchrono 

us 

discussion 

list 

Medical doctors 
(CPD); Norway 

Imprecision: size 
of the effect ranges 
from very small to 
large 

High risk of 
attrition and 
selection bias 

EBHC 
knowledge 
and skills 

1.40 

(-0.06 to 
2.85) 

n=i63 

2 studies 

- EBP 
training 
(face-to- 
face) plus 

EBP 

checklist 
and online 
support 

EBP course 

(face-to- 

face) with 

additional 

online 

learning 

Physiotherapists 

(CPD); 

Philippines 

Nurses (CPD); 
Spain 

High risk of 
selection and 
attrition bias 
Significant 
heterogeneity 
between non-RCT 
and RCT 

Although both 
studies show 
significant effects 
favouring blended 
learning, the 
pooled effect is 
non-significant 
with a very wide 
confidence interval 

EBHC 
knowledge 
and skills (1 
month post- 
intervention) 

0.90 
(0.42 to 
1.38) 

n=24i 

2 studies 

EBP course 

(face-to- 

face) with 

additional 

online 

learning 

Interactive 

e-learning 

course on 

EBM plus 

workshop 

(face-to- 

face) 

Nurses (CPD); 
Spain 

Medical doctors 

(CPD); 

Netherlands 

High risk of 
selection and 
attrition bias 
Significant 
heterogeneity 
between non-RCT 
and RCT 

Imprecision: effect 
ranges from small 
to very large 

EBHC 
knowledge 
and skills 
(3+ months 
post- 
intervention) 

1.11 

(0.80 to 
1.42) 

n=i86 

2 studies 

- EBP 
training 
(face-to- 
face) plus 

EBP 

checklist 

and online 

support 

Interactive 

e-learning 

course on 

EBM plus 

workshop 

(face-to- 

face) 

Physiotherapists 

(CPD); 

Philippines 
Medical doctors 
(CPD); 
Netherlands 

High risk of 
attrition bias 
Significant 
heterogeneity 

EBHC 

attitude 

0.17 

(-0.09 to 

0.43) 

n=226 

2 studies 

EBP course 
(face-to- 
face) with 

Nurses (CPD); 
Spain 

High risk for 
selection and 
attrition bias 
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additional 
online 
learning 
- EBM 
workshop 
with access 
to databases 
and 

asynchrono 

us 

discussion 

list 

Medical doctors 
(CPD); Norway 


EBHC 
attitude (l 
month post- 
intervention) 

0.05 
(-0.34 to 
0.44) 

n=24i 

2 studies 

EBP course 

(face-to- 

face) with 

additional 

online 

learning 

Interactive 

e-learning 

course on 

EBM plus 

workshop 

(face-to- 

face) 

Nurses (CPD); 
Spain 

Medical doctors 

(CPD); 

Netherlands 

High risk of 
selection and 
attrition bias 
Heterogeneity 
moderate 

EBHC 
attitude (3+ 
months 
post- 
intervention) 

0.32 

(-0.02 to 
O.67) 

n=i32 

1 study 

Interactive 

e-learning 

course on 

EBM plus 

workshop 

(face-to- 

face) 

Medical doctors 

(CPD); 

Netherlands 

High risk of 
attrition bias 

EBHC 

behaviour 

0.06 
(-0.28 to 
0.40) 

n=207 

2 studies 

EBP course 
(face-to- 
face) with 
additional 
online 
learning 
- EBM 
workshop 
with access 
to databases 
and 

asynchrono 

us 

discussion 

list 

Nurses (CPD); 
Spain 

Medical doctors 
(CPD); Norway 

High risk of 
selection and 
attrition bias 

EBHC 
behaviour 
(1 month 
post- 
intervention) 

0.19 

(-0.19 to 
0.56) 

n=i09 

1 study 

EBP course 

(face-to- 

face) with 

additional 

online 

learning 

Nurses (CPD); 
Spain 

High risk of 
selection bias, 
unclear risk of 
attrition bias 

EBHC 
behaviour 
(3+ months 
post- 
intervention) 

0.61 
(0.21 to 
1.01) 

n=ioo 

1 study 

Interactive 

e-learning 

course on 

EBM plus 

workshop 

(face-to- 

face) 

Medical doctors 

(CPD); 

Netherlands 

High risk of 
attrition bias 
Imprecision: 

Effect size ranges 
from small to large 
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12.3 PURE E-LEARNING VS FACE-TO-FACE LEARNING (6 STUDIES) 


Outcome 

SMD ( 95 % 
Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

knowledge 

-0.03 
(-0.26 to 
0.20) 

11=632 

5 studies 

Recorded 
PowerPoints 
on CD-ROM 
or available 
online 

One study 
had 

additional 

online 

exercises 

Medical doctors 
(UG); Norway, 

UK 

Medical doctors 
(CPD); UK 

Nurses (CPD); 
Japan 

High risk for 
attrition bias, 
unclear risk of 
selection bias 
Moderate 
heterogeneity (l 
study with high 
attrition rates in 
the intervention 
group is an 
outlier) 

EBHC 

skills 

-0.15 
(-0.34 to 
0.04) 

n=457 

2 studies 

Recorded 
PowerPoints 
on CD-ROM 
Online 
journal club 
with 

asynchrono 

us 

discussion 

group 

Medical doctors 
(UG); Norway 
Medical doctors 
(PG); USA 

High risk for 
detection and 
attrition bias, 
unclear risk of 
selection bias 

EBHC 

attitude 

0.11 

(-0.27 to 
0.48) 

n=in 

1 study 

Recorded 
PowerPoints 
on CD-ROM 

Medical doctors 
(UG); Norway 

Two studies did 
not report 
adequate data for 
attitude and were 
not included in the 
random-effects 
meta-analysis 


Return to text 


12.4 BLENDED LEARNING VS FACE-TO-FACE LEARNING (5 STUDIES) 


Outcome 

SMD 

(95% 

Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

knowledge 

0.28 

(- 

0.23 

to 

0.79) 

n=i46 

1 study 

Recorded 

PowerPoint 

presentations, 

online 

exercises, 

clinical 

activities, 

access to 

clinical tutor 

Medical 
doctors 
(PG); UK 
and 

Netherlands 

High risk of selection 
and attrition bias 
Imprecision: wide 95% 
confidence intervals 

One study did not 
report means and SDs, 
awaiting author 
response 

EBHC 
knowledge 
and skills 

-0.23 

(- 

0.52 

to 

0.06) 

n=i84 

2 studies 

Face-to-face 
workshop, self- 
directed, 
online 
learning, 
presentation of 

Medical 

doctors 

(UG); 

Australia 

and 

Malaysia 

High risk of selection, 
detection and attrition 
bias 

Only crude results 
reported for the cluster 
RCT. Results adjusted 
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patient-based 
EBM cases 
Lectures and 
small group 
activities in 
classroom, 
online lectures 
on YouTube, 
access to 
resources and 
library, mobile 
access to 
evidence at 
bedside 


by calculating effective 
sample size post-hoc 

EBHC 

skills 

-0.21 

(- 

0.68 

to 

0.26) 

n=yo 

1 study 

Online tutorial 
plus hands-on 
computer- 
based training 

Nurses 
(UG); UK 

Unclear risk of 
selection and detection 
bias 

Small sample size 

EBHC 

attitude 

1.07 

( 0.57 

to 

1.58) 

11=69 

1 study 

Lectures and 
small group 

activities in 

classroom, 
online lectures 
on YouTube, 
access to 

resources and 
library, mobile 
access to 

evidence at 

bedside 

Medical 

doctors 

(UG); 

Australia 

and 

Malaysia 

High risk of attrition 
bias, unclear risk of 
selection bias 

Only crude results 
reported for the cluster 
RCT. Results adjusted 
by calculating effective 
sample size post-hoc 
Only 17% (44/263) of 
participants from the 
blended learning group 
and 16% (35/234) of 
the face-to-face 
learning group 
completed the 
questionnaire 
Imprecision: Effect size 
ranges from medium to 
very large 

Two studies did not 
report data sufficiently 
to be included in the 
meta-analysis 

EBHC 

behaviour 

2-34 

(1.72 

to 

2.96) 

n=69 

1 study 

Lectures and 
small group 

activities in 

classroom, 
online lectures 
on YouTube, 
access to 

resources and 
library, mobile 
access to 

evidence at 

bedside 

Medical 

doctors 

(UG); 

Australia 

and 

Malaysia 

High risk of attrition 
bias, unclear risk of 
selection bias 

Only crude results 
reported for the cluster 
RCT. Results adjusted 
by calculating effective 
sample size post-hoc 
Only 17% (44/263) of 
participants from the 
blended learning group 
and 16% (35/234) of 
the face-to-face 
learning group 
completed the 
questionnaire 
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12.5 BLENDED LEARNING VS PURE E-LEARNING (3 STUDIES) 


Outcome 

SMD 

( 95 % 

Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

knowledge 

0.69 

(0.40 

to 

0.99) 

n=i 93 

2 studies 

Access to online 
learning site plus 
face-to-face small 
group discussions vs 
Access to online 
learning site, online 
exercises and 
asynchronous 
discussion list 
Recorded 

PowerPoint 
presentations plus 
clinical activities 
and access to 
clinical tutor vs 
recorded 

PowerPoint and 
access to online 
facilitator on 
demand 

Physician 

assistants 

(UG); 

USA 

Medical 

doctors 

(PG); 

various 

LMICs 

Unclear risk of 
selection and 
attrition bias 

High risk of 
recruitment bias 
and loss of clusters 
for the cluster RCT 
Size of the effect 
ranges from small 
to large 

EBHC 

skills 

-0.53 

(-2.31 

to 

1-25) 

n= 2 i 8 

2 studies 

Computer-lab 
teaching plus access 
to online learning 
site vs DVD 
containing recorded 
PowerPoints and 
tutorials and access 
to online learning 
material 

Recorded 

PowerPoint 
presentations plus 
clinical activities 
and access to 
clinical tutor vs 
recorded 

PowerPoint and 
access to online 
facilitator on 
demand 

Nurses 

(PG); 

Australia 

and Hon 

Kong 

Medical 

doctors 

(PG); 

various 

LMICs 

High risk for 
selection and 
attrition bias for 
non-RCT 

High risk of 
recruitment bias 
and loss of clusters 
for the cluster RCT 
Inconsistency: 
significant 
heterogeneity 
between studies. 
Results favour 
pure e-learning 
for the non-RCT 
and blended 
learning for the 
cluster RCT 
Imprecision: very 
wide confidence 
intervals - results 
favour pure e- 
learning for the 
non-RCT and 
blended learning 
for the cluster RCT 


Return to text 
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12.6 PURE E-LEARNING VS PURE E-LEARNING (3 STUIDES) 


Outcome 

SMD 

(95% 

Cl) 

No of 

participants 

(studies) 

Interventions 

Participants 
(level of 
education); 
Country 

Limitations 

EBHC 

skills 

1.30 

(0.68 

to 

1-93) 

n=H9 

2 studies 

DVD containing 
recorded 

PowerPoints and 
tutorials and access 
to online learning 
material vs standard 
distance learning 
(online) 

Online j ournal club 
with asynchronous 
discussion list vs 
emails containing 
the articles plus 
access to journals 

Nurses 

(PG); 

Australia 

and Hon 

Kong 

Medical 

doctors 

(CPD); 

Canada 

High risk of 
selection, attrition 
and detection bias 
Imprecision: effect 
size ranges from 
medium to very 
large 

High levels of 
heterogeneity 

One study did not 
report data in a 
way that it could 
be incorporated in 
the meta-analysis 


Return to text 
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13 Differences between protocol and 
review 


13.1 OUTCOMES 


In the protocol, we pre-specified the following primary outcomes: 

1. EBHC knowledge 

2. EBHC skills 

3. EBHC attitudes 

4. EBHC behaviour 

In the review, we added an extra outcome “EBHC knowledge and skills”, since many of the 
included trials measured this as a composite outcome. Therefore, the primary outcomes in the 
review are as follows: 

1. EBHC knowledge 

2. EBHC knowledge and skills 

3. EBHC skills 

4. EBHC attitudes 

5. EBHC behaviour 


13.2 SUBGROUP ANALYSIS 


In the protocol, we pre-specified the following subgroup analysis: 

1. Type of healthcare professional (e.g. medical vs allied healthcare professionals) 

2. Level of education of healthcare professionals (undergraduate vs postgraduate vs continuing 
medical education) 

3. Measurement tool used for outcomes (e.g. Fresno test vs Berlin test) 

4. Synchronicity (synchronous vs asynchronous delivery) 

5. Duration of the intervention 

In the review, due to the limited number of included studies per outcome, we were only able to 
conduct the following subgroup analysis: 

1. Study design (non-RCTs vs RCTs) 
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w CampbellCollaboration 


About this review 

Evidence-based health care (EBHC) involves phrasing questions based on a knowledge gap, 
searching for research that can answer the question, critically appraising and interpreting the 
research, applying the results and auditing the process. Electronic learning (e-learning) has 
become an increasingly popular method of teaching EBHC. 

This review assesses the effectiveness of e-learning of EBHC for increasing EBHC 
competencies in healthcare professionals. The primary outcomes are EBHC knowledge, skills, 
attitude and behaviour. E-learning, compared to no learning, improves EBHC knowledge and 
skills but not attitudes and behaviour. There is no difference in outcomes when comparing 
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